Keywords: Human Factors and Human-in-the-Loop, Human-Centered Robotics, Human-Centered Automation
Abstract: Collaborative robots have been widely used in emerging working contexts such as smart manufacturing. How to make the robot understand human factors (HF, e.g., trust, comfort, acceptance) still remains a gap in the research community. In this Late Breaking poster presentation, we will show the preliminary results of our work in human factors characterization and robot task scheduling in human-robot collaborative tasks. We found that human factors-related physiological and physical information can be used to characterize the HF levels in human-robot partnerships. This work has the potential to improve human-robot collaboration safety and productivity in smart manufacturing contexts.

Keywords: Imitation Learning, Neural and Fuzzy Control, Learning from Demonstration
Abstract: Imitation learning allows robots to acquire complex actions by imitating expert demonstrations, removing the need for manual control policy design or reward shaping. It has been successfully applied in manipulation, locomotion, and navigation tasks. However, existing approaches often rely on deep neural networks, which require large amounts of expert data and suffer from training instability, high computational costs, and poor interpretability. We propose a fuzzy rule-based imitation learning method that generates interpretable IF-THEN rules from expert demonstrations. Using the Mamdani product and a weighted average mechanism, actions are produced with fewer learnable parameters. This improves learning efficiency, enhances transparency, and enables dynamic adaptation to new conditions without additional expert data. The approach is validated on a 7-DoF robotic manipulator, showing robust and generalizable action generation under varied scenarios.

Keywords: Data Sets for Robot Learning, Model Learning for Control, Motion Control
Abstract: In this paper, a novel and efficient non-parametric dynamics modeling method, called EffiDynaMix, is proposed. This gray-box, data-driven technique combines the Gaussian Mixture Model (GMM) with the mathematical inertia model of robot behavior. This integration simplifies the creation of train datasets and enhances the model's generalization capabilities. Extensive testing via simulations and experiments on a robot manipulator indicates that EffiDynaMix outperforms conventional data-driven methods such as non-Optimal GMM (nonOpt-GMM), chirp trained GMM (chirp-GMM), Sparse Gaussian Processes (SGP), Long Short-Term Memory (LSTM) networks, and Radial Basis Function Neural Networks (RBFNN) in both training efficiency and accuracy, particularly in new data scenarios. By incorporating system dynamics, EffiDynaMix not only streamlines the learning process but also ensures smooth adaptation to unfamiliar situations.

Keywords: Aerial Systems: Mechanics and Control, Robust/Adaptive Control, Aerial Systems: Applications
Abstract: Flapping-wing aerial manipulators offer promising capabilities for efficient and lightweight robotic applications but remain highly susceptible to modeling uncertainties and external disturbances due to their nonlinear, time-varying dynamics and low Reynolds number operation. These challenges significantly hinder their ability to perform stable manipulation tasks. Although robust control presents a viable solution, its implementation is nontrivial for flapping-wing platforms due to complex dynamics and susceptibility to uncertainties and external disturbances. This work proposes a parallel nonlinear disturbance observer-based (NDOB) robust controller targeting the altitude and longitudinal dynamics of an X-shaped flapping-wing UAV. The controller is designed to suppress undesired motion during manipulation, improving stability and disturbance rejection. Experimental results on the Flapper Nimble+ platform validate the effectiveness of the proposed method, demonstrating accurate trajectory tracking and robustness under wind disturbances and varying payloads.

Keywords: Rehabilitation Robotics, Design and Human Factors, Perception-Action Coupling
Abstract: Functional Movement Disorders (FMD) involve impaired voluntary motor control despite no structural neurological damage and are increasingly understood within a predictive coding framework. Disruptions in sensory attenuation and prediction error updating impair the sense of agency in FMD. We present an eXtended Reality (XR)-based biofeedback platform that integrates haptic exafference, cued motor imagery, and adaptive reinforcement to recalibrate sensorimotor control through operant conditioning. Using a Kalman filter and Linear-Quadratic-Gaussian (LQG) control model, we simulate disrupted internal modeling in FMD and demonstrate how motor imagery suggestions for preconditioning and haptic postconditioning can restore control by enhancing state estimation and reducing prediction error. Our co-designed platform incorporates real-time visual and force feedback, and wearable brain imaging to monitor neural correlates of agency. Usability studies and simulation results highlighted the potential of XR-based adaptive control for personalized rehabilitation in FMD, bridging computational neuroscience, robotics, and clinical neurorehabilitation.

Keywords: Perception-Action Coupling, Surgical Robotics: Planning, Sensorimotor Learning
Abstract: Understanding how the brain transitions between cognitive states during complex tasks is a central challenge in neuroscience for neuroengineering. This study investigates EEG microstate dynamics during the “suturing and intracorporeal knot-tying” task—the most complex skill in the Fundamentals of Laparoscopic Surgery (FLS) program. Using multimodal neuroimaging data (EEG and fNIRS), we examine expertise-dependent brain activation patterns through the lens of a Kalman Filter-based human motor control framework. Microstates—quasi-stable spatial EEG patterns—are mapped to functional brain networks and modeled as elements of internal state prediction, sensory feedback, and error correction. Six EEG-fNIRS microstates were extracted using K-Means clustering and aligned with canonical microstates (A–F), capturing transitions linked to salience, attention, and motor planning. Experts demonstrated efficient microstate transitions during both task initiation and error response, engaging networks like the frontoparietal, salience, and default mode networks, while novices exhibited variable and reactive patterns. Kalman-based control model provided a unified framework linking neurophysiological microstates with adaptive motor control, highlighting how experience shapes cognitive and sensorimotor dynamics during skill learning.

Keywords: Deep Learning in Grasping and Manipulation
Abstract: Reliable object grasping is one of the fundamental tasks in robotics. However, determining grasping pose based on single-image input has long been a challenge due to limited visual information and the complexity of real-world objects. In this work, we propose Triplane Grasping, a fast grasping decision-making method that relies solely on a single RGB-only image as input. Triplane Grasping creates a hybrid Triplane-Gaussian 3D representation through a point decoder and a triplane decoder, which produce an efficient and high-quality reconstruction of the object to be grasped to meet real-time grasping requirements. We propose to use an end-to-end network to generate 6-DoF parallel-jaw grasp distributions directly from 3D points in the point cloud as potential grasp contacts and anchor the grasp pose in the observed data. Experiments on the OmniObject3D and GraspNet-1Billion datasets demonstrate that our method achieves rapid modeling and grasping pose decision-making for daily objects, and strong generalization capability.

Keywords: Grasping, Performance Evaluation and Benchmarking, Deep Learning in Grasping and Manipulation
Abstract: The Collaborative Open-source Manipulation Performance Assessment for Robotics Enhancement (COMPARE) Ecosystem aims to create a greater cohesion between open-source products that improves the modularity of open-source components in the robot manipulation pipeline for more effective performance benchmarking through the generation of community-driven standards for components of software pipelines, benchmarking practices, objects, datasets, and hardware designs. Online resources for COMPARE include robot-manipulation.org — a landing page for open-source and benchmarking in robot manipulation, including repositories of open-source products and benchmarking assets, event listings for workshops, competitions, and webinars, leaderboards of benchmarking results from across the community, and more — and the COMPARE Slack workspace — the primary online discussion platform for COMPARE, with channels corresponding to each of the relevant open-source and benchmarking thrusts. As part of activating the COMPARE Ecosystem, we are holding a full-day workshop at RSS 2025 — Benchmarking Robot Manipulation: Improving Interoperability and Modularity — and hosting a 1-week COMPARE Summer School Cooperative Hackathon and Benchmarking Exercise. This poster provides an overview of the COMPARE Ecosystem project, information about upcoming events, available resources, and a call for engagement with the international robot manipulation community.

Keywords: Computer Vision for Medical Robotics, Visual Tracking, Deep Learning Methods
Abstract: Ultrasound (US)-guided needle insertion is crucial for minimally invasive interventions, yet robust needle tracking remains challenging due to the scarcity of labeled training data. This paper introduces a data-efficient fine-tuning (DEFT) framework incorporating a motion prefix and tunable register to reduce data dependency on labeled US needle tracking dataset while maintaining accuracy. With the proposed plug-and-play motion prefix, it enables task-oriented adaptation of historical motion at minimal dependency on training data. By further incorporating the lightweight tunable register, the global context can be adapted to small US datasets without losing learned behavior from pretraining domain. Evaluations were conducted on collected datasets with motorized and manual insertions using different US machines. By pretraining on large-scale open-world datasets and fine-tuning on limited US data, our model demonstrates state-of-the-art performance on various data-insufficient or out-of-distribution scenarios. To the best of our knowledge, this is the first US needle tracker trained with DEFT, offering a practical solution for clinical deployment where labeled data is scarce.

Keywords: Vision-Based Navigation, Data Sets for Robot Learning, Imitation Learning
Abstract: Visual Semantic Navigation (VSN) is a key problem in robotics where an agent must find a target object in an unknown environment using mostly visual cues. Most current VSN models are trained in simulated environments, often using rendered scenes that resemble the real world. These models generally rely on raw RGB images, which limits their ability to generalize to real-world scenarios due to domain adaptation issues.

To address this, we propose SemNav, a novel approach that uses semantic segmentation as the main visual input. By incorporating high-level semantic information, our model enhances the agent’s perception and decision-making abilities. This allows it to learn more robust navigation policies that generalize better to unseen environments, both in simulation and in the real world.

We also introduce a new dataset—the SemNav dataset—specifically designed to train navigation models that use semantic segmentation. We evaluate our method extensively in simulated environments (using Habitat 2.0 and the HM3D dataset) and with real-world robotic platforms.

Experiments show that SemNav outperforms current state-of-the-art VSN methods, achieving higher success rates. Additionally, our real-world tests confirm that semantic segmentation significantly helps bridge the sim-to-real gap, making our method a strong candidate for practical VSN applications. We will release the dataset, code, and trained models to the public.

Keywords: Marine Robotics, Legged Robots, Machine Learning for Robot Control
Abstract: Precise close-contact inspections are critical in underwater environments, where complex dynamics and biofouling pose significant challenges to traditional vehicles. To address these issues, this study develops a Reinforcement Learning-based framework to optimize the gait of an underwater legged robot for accurate and stable locomotion. The simulation environment integrates hydrodynamic forces, buoyancy effects, added mass, and seabed interactions to train robust gait control policies. By designing the action space to regulate phase and amplitude differences within a legged oscillation model, the framework ensures predictable gait patterns. Experimental validation was conducted both in simulation and on a real robot, confirming that the RL-trained policy effectively mitigates hydrodynamic disturbances. The results demonstrated reduced altitude and pitch instability during fast forward walking, as well as improved heading accuracy in curved trajectories.

Keywords: Human-Aware Motion Planning, Planning under Uncertainty, Social HRI
Abstract: Human motion is stochastic and ensuring safe robot navigation in a pedestrian-rich environment requires proactive decision-making. Past research relied on incorporating deterministic future states of surrounding pedestrians which can be overconfident leading to unsafe robot behaviour. The current paper proposes a predictive uncertainty-aware planner that integrates neural network based probabilistic trajectory prediction into planning. Our method uses a deep ensemble based network for probabilistic forecasting of surrounding humans and integrates the predictive uncertainty as constraints into the planner. We compare numerous constraint satisfaction methods on the planner and evaluated its performance on real world pedestrian datasets. Further, offline robot navigation was carried out on out-of-distribution pedestrian trajectories inside a narrow corridor.

Keywords: Human-Centered Robotics, Human-Robot Collaboration
Abstract: Deep reinforcement learning (DRL) is one of the methods used to achieve autonomous mobility in robots. Although it effectively achieves high performance, there is still a problem that the decision-making process of robot actions is unclear. To address this, many studies have been conducted to enable visual analysis through images. However, these images lack a direct connection to physical space, which limits user understanding of robot actions. Therefore, we present an encoder-decoder model based on Transformer that enables a visual explanation of DRL-based robot autonomous movement and projects it onto physical space using augmented reality(AR). In this paper, we demonstrate that the proposed method provides a direct visual explanation for different robot actions and that the visualization using AR effectively enhances users’ understanding of robot actions.

Keywords: Soft Sensors and Actuators, Actuation and Joint Mechanisms, Soft Robot Materials and Design
Abstract: Conventional robot actuation often relies on more actuators for greater dexterity, complicating design and control. While soft robots with passive under-actuation or multi-stable metamaterials simplify actuation, they rely on external forces or limited multi-stable structures. Here, we introduce a Hysteresis-Assisted Shape-Morphing (HasMorph) paradigm for soft continuum robots and integrate it with tip-everting soft-growing robots (SGRs) to enable scalable shape-morphing in confined spaces. HasMorph exploits shape hysteresis—differences in configuration during loading and unloading—to achieve multi-section bending with minimal actuators through an Inverted Zigzag Tendon-Sheath Mechanism. By alternating just two actuators, this mechanism enables billions of stable shape changes in soft continuum robots, vastly expanding their configuration space and workspace without added hardware complexity. By leveraging friction-induced hysteresis, the SGRs can reversibly morph its shape while maintaining frictionless tip growth. Experiments were conducted to demonstrate reversible shape morphing, tip steering, and follow-the-leader tip growth in unstructured spaces. These findings underscore the potential of hysteresis-assisted actuation for dexterous manipulation and navigation of soft continuum robots, paving the way for advanced applications in medical intervention, industrial inspection, and exploration.

Keywords: Deep Learning Methods, Representation Learning, Machine Learning for Robot Control
Abstract: As autonomous systems are increasingly deployed in open and uncertain settings, there is a growing need for trustworthy world models that can reliably predict future high-dimensional observations. The learned latent representations in world models lack direct mapping to meaningful physical quantities and dynamics, limiting their utility and interpretability in downstream planning, control, and safety verification. In this paper, we argue for a fundamental shift from physically informed to physically interpretable world models --- and crystallize four principles that leverage symbolic knowledge to achieve these ends: (1) structuring latent spaces according to the physical intent of variables, (2) learning aligned invariant and equivariant representations of the physical world, (3) adapting training to the varied granularity of supervision signals, and (4) partitioning generative outputs to support scalability and verifiability. We experimentally demonstrate the value of each principle on two benchmarks. This paper opens several intriguing research directions to achieve and capitalize on full physical interpretability in world models.

Keywords: Sensor Fusion, Object Detection, Segmentation and Categorization, Computer Vision for Transportation
Abstract: This paper proposes a Bird’s Eye View (BEV)- based camera-LiDAR fusion algorithm to enhance long-range 3D object detection by leveraging LiDAR-to-image depth projection and voxel-wise average pooling. Accurate detection of distant objects is essential for autonomous driving safety, yet existing BEV-based fusion methods suffer from inaccurate depth estimation and imbalanced feature intensities. To address these limitations, the proposed method leverages accurate depth maps generated by projecting LiDAR points onto images for accurate geometric transformations of camera features into 3D space, and employs voxel-wise average pooling to balance feature intensities across distances, producing consistent BEV representations. Evaluations on the nuScenes dataset, using a distance-scaled linear matching threshold, demonstrate significant improvements, notably a 49.2% of mean average precision increase for objects beyond 60 m.

Keywords: Compliant Assembly, Force and Tactile Sensing, Compliant Joints and Mechanisms
Abstract: This paper presents the design and experimental validation of a torque-controlled robotic wrist with high torque density. The proposed wrist features a serial pitch-yaw joint configuration that enhances dexterity while maintaining compactness. The design integrates stepper motors, harmonic geartrains, and compliant mechanisms to optimize torque output and control accuracy. A compliant pulley and a compliant cap are introduced, enabling embedded torque sensing without the need for external sensors, thereby reducing system complexity and improving response time. Experimental results demonstrate the effectiveness of the wrist in torque accuracy, backdrivability, impedance regulation, and misalignment correction during peg-in-hole assembly, highlighting the benefits of the compliant-driven torque sensing approach. Compared to existing robotic wrists, the proposed design achieves a higher torque density. The findings contribute to advancing robotic wrist technology, particularly in applications requiring precise force modulation, high dexterity, and adaptable compliance.

Keywords: Dual Arm Manipulation, Manipulation Planning
Abstract: In this paper, we address the problem of manipulating multi-particle aggregates using a bimanual robotic system. Our approach enables the autonomous transport of dispersed particles through a series of shaping and pushing actions using robotically-controlled tools. Achieving this advanced manipulation capability presents two key challenges: high-level task planning and trajectory execution. For task planning, we leverage Vision Language Models (VLMs) to enable primitive actions such as tool affordance grasping and non-prehensile particle pushing. For trajectory execution, we represent the evolving particle aggregate's contour using truncated Fourier series, providing efficient parametrization of its closed shape. We adaptively compute trajectory waypoints based on group cohesion and the geometric centroid of the aggregate, accounting for its spatial distribution and collective motion. Through real-world experiments, we demonstrate the effectiveness of our methodology in actively shaping and manipulating multi-particle aggregates while maintaining high system cohesion.

Keywords: Modeling, Control, and Learning for Soft Robots, Soft Robot Applications
Abstract: State estimation for soft continuum robots is challenging due to their infinite-dimensional states (poses, strains, velocities) resulting from continuous deformability, while conventional sensors provide only discrete data. A recent method, called a boundary observer, uses Cosserat rod theory to estimate all robot states by measuring only tip velocity. In this work, we propose a novel boundary observer that instead measures the internal wrench at the robot’s base, leveraging the duality between velocity and internal wrench. Both observers are inspired by energy dissipation, but the base-based approach offers a key advantage: it uses only a 6-axis force/torque sensor at the base, avoiding the need for external sensing systems. Combining tip- and base-based methods further enhances energy dissipation, speeds up convergence, and improves estimation accuracy. We validate the proposed algorithms in experiments where all boundary observers converge to the ground truth within 3 seconds, even with large initial deviations, and they recover from unknown disturbances while effectively tracking high-frequency vibrations.

Keywords: Visual-Inertial SLAM, Deep Learning for Visual Perception, Localization
Abstract: Visual-based localization has made significant progress, yet its performance often drops in large-scale, outdoor, and long-term settings due to factors like lighting changes, dynamic scenes, and low-texture areas. These challenges degrade feature extraction and tracking, which are critical for accurate motion estimation. While learning-based methods such as SuperPoint and SuperGlue show improved feature coverage and robustness, they still face generalization issues with out-of-distribution data. We address this by enhancing deep feature extraction and tracking through self-supervised learning with task-specific loss function design. Our method promotes stable and informative features, improving generalization and reliability in challenging environments.

Keywords: Prosthetics and Exoskeletons, Machine Learning for Robot Control, Rehabilitation Robotics
Abstract: Current exoskeleton controllers are constrained to predefined, cyclic tasks (e.g., flat-ground walking) and struggle to adapt to the variability of real-world environments, with only 4% of rehabilitation studies validated in home/community settings. This limitation poses safety risks, as preprogrammed torque profiles may fail during novel movements. We propose a deep learning-based framework to enable task-agnostic control, a high level uncertainty classifier-based controller. The classifier detects whether user actions align with "in-distribution" trained tasks (walking, jogging on inclines/declines) or represent novel "out-of-distribution" scenarios, switching between task-specific assistance and safety-optimized torque. Leveraging biomechanical time-series data from the Dephy Exoboot, three temporal convolutional network (TCN) based architectures—ensembles, autoencoders, and GANs—were trained on actuated/unactuated locomotion data. Evaluated offline on a test set (N=3 users) containing trained tasks (treadmill/overground walking/jogging) and novel actions (stairs, jumping, sitting), the model achieved an F1-score of 0.99 and a J-statistic of 0.985. Results demonstrated clear separation between in-distribution (assisted) and out-of-distribution (safety mode) tasks, even during dynamic transitions. Future work will validate the system in online, continuous environments outside the lab, advancing toward deployable exoskeletons for diverse daily activities.

Keywords: Autonomous Vehicle Navigation, SLAM
Abstract: Autonomous docking is a crucial capability for achieving long-term and ultimate autonomy in unmanned vehicle operations. Monocular cameras can be considered as one of the most efficient and promising sensing options for autonomous docking because many existing unmanned vehicles have at least one onboard camera for teleoperation or monitoring. However, monocular cameras possess a fundamental limitation as a stand-alone position-sensing device in the state estimation for autonomous docking due to their nature of bearings-only sensing and the lack of range observability. This study proposes an autonomous docking method for nonholonomic vehicles using monocular vision-based bearing-only measurements. The proposed approach computes the control inputs that enable docking while drawing trajectories that simultaneously consider the vehicle’s motion constraints and state observability. Simulation results are shown to validate the proposed method under vehicle docking scenarios with a single-point landmark for the dock and bearing-only sensing.

Keywords: Prosthetics and Exoskeletons, Tendon/Wire Mechanism, Underactuated Robots
Abstract: Structural lightness and functional versatility are key requirements in robotic prosthetic hands. To achieve both characteristics, this study proposes a novel underactuated thumb mechanism that enables independent 2-DOF motion using a single actuator. The design employs the Geneva mechanism’s intermittent motion to control ab/adduction, while its non-engaged range of motion drives flexion/extension via a wire pulley system. A shape-adaptive grasp mechanism is also implemented to enable both power and pinch grips. The mechanism achieves independent motion on different planes with minimal components. Experimental results demonstrate that the proposed system effectively realizes 2-DOF thumb motion using only one actuator, offering a compact and modular solution for next-generation prosthetic hands.

Keywords: Search and Rescue Robots, Aerial Systems: Applications, Aerial Systems: Perception and Autonomy
Abstract: In mountain rescue operations, unmanned aerial vehicles (UAVs) must repeatedly ascend and descend. However, most UAVs face limitations on their vertical descent speed. Exceeding this limit can trigger a dangerous airflow condition known as the vortex ring state (VRS). The VRS causes the UAV to lack rotor speed response and increases fuselage vibration, making it uncontrollable. Howeverpilots or operators cannot perceive the onset of VRS, the timing of exiting the VRS is critical yet challenging. To address these issues, we propose a strategy allowing UAVs to maintain controlled descent within the VRS or recover to a hover state. We developed a VRS vibration model by analyzing the relationship between descent rate, rotor speed, and vibration to calculate the vibration under VRS.Computational Fluid Dynamics (CFD) simulations were employed to confirm VRS flowfield and validate the vibration model. Then we take the realflight tests, we descended the UAV at 10 m/s, recording the descent rate and vibration amplitude with Inertial Measurement Unit (IMU). We assume the high-speed descent UAV can detect and respond to VRS and apply Support Vector Machine (SVM) techniques to analyze vibration signal changes, distinguishing VRS characteristics from those observed during hover and climb phases in real-flight data. With this approach verified, we further tested high-speed descents at 12 m/s and 15 m/s. This study reveals the function between VRS vibration, rotor speed,

Keywords: Mobile Manipulation
Abstract: Optimal base placement is a critical problem in mobile manipulation, as the position of the mobile base directly influences task feasibility and manipulability. In this work, we propose a real-time base placement method using a 3D Inverse Reachability Map (IRM). By slicing the 3D IRM into 2D layers and leveraging manipulability-based metrics, our method efficiently identifies optimal base poses and trajectories for multiple task goals. The proposed approach supports dynamic task scenarios and outperforms prior methods in computation time. Experimental results using a 7-DOF Franka Emika Panda robot demonstrate the effectiveness of the method for both static task poses and continuous end-effector trajectories.

Keywords: AI-Enabled Robotics, Object Detection, Segmentation and Categorization, Deep Learning in Grasping and Manipulation
Abstract: Achieving robust robotic intelligence for real-world tasks remains a challenge due to the need for precise control and contextual adaptability. This work introduces an integrative AI framework that synergizes GPT-4’s contextual reasoning with reinforcement learning (RL) for enhanced task planning and object manipulation. Using Hello Robot’s Stretch 2, our approach interprets natural language instructions, generates strategic plans, and executes actions in dynamic environments. Experiments across static, dynamic, and complex scenarios demonstrate superior performance, with success rates of 90%, 85%, and 80%, respectively, outperforming standalone RL, GPT-4, and state-of-the-art methods like TidyBot (85%) and OK-Robot (75%). Our framework bridges high-level reasoning and low-level control, reducing completion times and retries, and paves the way for more autonomous, efficient robotic systems.

Keywords: Medical Robots and Systems, Mechanism Design, Surgical Robotics: Planning
Abstract: Trauma to the ankle is a common injury and a major source of long-term disability. Each year, over half a million ankle injuries require surgical intervention. In cases involving syndesmotic sprains, surgical manipulation of the tibia and fibula is necessary to properly align and reduce the syndesmosis space. However, current manual reduction techniques—whether open or percutaneous—often result in inaccurate reduction. We propose a novel system that integrates intraoperative low-dose cone-beam computed tomography (CBCT) with 3D-2D image registration and robotic manipulation of the fibula to precisely restore its anatomical alignment with the tibial incisura, while minimizing radiation exposure to both patients and surgical staff. This study reports on the development of the robotic assistant and the accompanying image-based 3D planning and guidance method. Experiments were conducted to evaluate the positioning accuracy of the robotic system and the accuracy of the multi-body 3D-2D registration. The results demonstrate that the robotic system can achieve the required motion with a free-space positioning accuracy of 0.26 ± 0.10 mm. The 3D-2D registration method achieved high accuracy of 0.2 ± 0.8 mm, enabling tracking of the robot and bones to dynamically correct the trajectory during the procedure.

Keywords: Human-Robot Teaming
Abstract: Situation awareness is important for decision making and performance in human-robot teams, yet both estimating and improving situation awareness in real-time is an open research area. We present a situation awareness system with a real-time situation awareness estimator and adaptive robot interventions. The effectiveness of the situation awareness system in improving situation awareness and performance is demonstrated through a human-robot teaming experiment.

Keywords: Reinforcement Learning, Deep Learning Methods, Physical Human-Robot Interaction
Abstract: Achieving human-level performance on real world tasks is a north star for the robotics community. We present the first learned robot agent that reaches amateur human-level performance in competitive table tennis. Table tennis is a physically demanding sport that takes humans years to master. We contribute (1) a hierarchical and modular policy architecture consisting of (i) low level controllers with their skill descriptors that model their capabilities and (ii) a high level controller that chooses the low level skills, (2) techniques for enabling zero-shot sim-to-real and curriculum building, including an iterative approach (train in sim, deploy in real), and (3) real time adaptation to unseen opponents. Policy performance was assessed through 29 robot vs. human matches of which the robot won 45% (13/29). All humans were unseen players and their skill level varied from beginner to tournament level. Whilst the robot lost all matches vs. the most advanced players it won 100% matches vs. beginners and 55% matches vs. intermediate players, demonstrating solidly amateur human-level performance.

Keywords: Big Data in Robotics and Automation, Methods and Tools for Robot System Design, Engineering for Robotic Systems
Abstract: Recent work suggests that very large datasets of teleoperated robot demonstrations can train transformer-based models that have the potential to generalize to new scenes, robots, and tasks. However, curating, distributing, and loading large datasets of robot trajectories, which typically consist of video, textual, and numerical modalities - including streams from multiple cameras - remains challenging. We propose Robo-DM, an efficient cloud-based data management toolkit for collecting, sharing, and learning with robot data. With Robo-DM, robot datasets are stored in a self-contained format with Extensible Binary Meta Language (EBML). Robo-DM reduces the size of robot trajectory data, transfer costs, and data load time during training. In particular, compared to the RLDS format used in OXE datasets, Robo-DM’s compression saves space by up to 70x (lossy) and 3.5x (lossless). Robo-DM also accelerates data retrieval by load-balancing video decoding with memory-mapped decoding caches. Compared to LeRobot, a framework that also uses lossy video compression, Robo-DM is up to 50x faster. In fine-tuning Octo, a transformer-based robot policy with 73k episodes with RT-1 data, Robo-DM does not incur any loss in training performance. We physically evaluate a model trained by Robo-DM with lossy compression, a pick-and-place task, and In-Context Robot Transformer. Robo-DM uses 75x compression of the original dataset and does not suffer any reduction in downstream task accuracy. Code and evaluation scripts can be found on the website.

Keywords: Integrated Planning and Control, Reactive and Sensor-Based Planning, Optimization and Optimal Control
Abstract: We introduce a novel gradient-based approach for solving sequential tasks by dynamically adjusting the underlying myopic potential field in response to feedback and the world's regularities. This adjustment implicitly considers subgoals encoded in these regularities, enabling the solution of long sequential tasks, as demonstrated by solving the traditional planning domain of Blocks World—without any planning. Unlike conventional planning methods, our feedback-driven approach adapts to uncertain and dynamic environments, as demonstrated by one hundred real-world trials involving drawer manipulation. These experiments highlight the robustness of our method compared to planning and show how interactive perception and error recovery naturally emerge from gradient descent without explicitly implementing them. This offers a computationally efficient alternative to planning for a variety of sequential tasks, while aligning with observations on biological problem-solving strategies.

Keywords: Deep Learning Methods, Transfer Learning
Abstract: In recent years, Embodied Artificial Intelligence (Embodied AI) has advanced rapidly, yet the increasing size of models conflicts with the limited computational capabilities of Embodied AI platforms. To address this challenge, we aim to achieve both high model performance and practical deployability. Specifically, we focus on Vision-and-Language Navigation (VLN), a core task in Embodied AI. This paper introduces a two-stage knowledge distillation framework, producing a student model, MiniVLN, and showcasing the significant potential of distillation techniques in developing lightweight models. The proposed method aims to capture fine-grained knowledge during the pretraining phase and navigation-specific knowledge during the fine-tuning phase. Our findings indicate that the two-stage distillation approach is more effective in narrowing the performance gap between the teacher model and the student model compared to single-stage distillation. On the public R2R and REVERIE benchmarks, MiniVLN achieves performance on par with the teacher model while having only about 12% of the teacher model's parameter count.

Keywords: Force and Tactile Sensing, Sensorimotor Learning, Learning from Demonstration
Abstract: Achieving robust dexterous manipulation in unstructured domestic environments remains a significant challenge in robotics. Even with state-of-the-art robot learning methods, haptic-oblivious control strategies (i.e. those relying only on external vision and/or proprioception) often fall short due to occlusions, visual complexities, and the need for precise contact interaction control. To address these limitations, we introduce PolyTouch, a novel robot finger that integrates camera-based tactile sensing, acoustic sensing, and peripheral visual sensing into a single design that is compact and durable. PolyTouch provides high-resolution tactile feedback across multiple temporal scales, which is essential for efficiently learning complex manipulation tasks. Experiments demonstrate an at least 20-fold increase in lifespan over commercial tactile sensors, with a design that is both easy to manufacture and scalable. We then use this multi-modal tactile feedback along with visuo-proprioceptive observations to synthesize a tactile-diffusion policy from human demonstrations; the resulting contact-aware control policy significantly outperforms haptic-oblivious policies in multiple contact-aware manipulation policies. This paper highlights how effectively integrating multi-modal contact sensing can hasten the development of effective contact-aware manipulation policies, paving the way for more reliable and versatile domestic robots. More information can be found at https://polytouch.alanz.info.

Keywords: Omnidirectional Vision, Visual Tracking, Aerial Systems: Applications
Abstract: In this paper, we present a new compact vision sensor consisting of two fisheye event cameras mounted back-to-back, which offers a full 360-degree view of the surrounding environment. We describe the optical design, projection model and practical calibration using the incoming stream of events, of the novel stereo camera, called SFERA. The potential of SFERA for real-time target tracking is evaluated using a Bayesian estimator adapted to the geometry of the sphere. Real-world experiments with a prototype of SFERA, including two synchronized Prophesee EVK4 cameras and a DJI Mavic Air 2 quadrotor, show the effectiveness of the proposed system for aerial surveillance.

Keywords: Marine Robotics, Environment Monitoring and Management, Field Robots
Abstract: Scalar field features such as extrema, contours, and saddle points are essential for applications in environmental monitoring, search and rescue, and resource exploration. Traditional navigation methods often rely on predefined trajectories, leading to inefficient and resource-intensive mapping. This paper introduces a new adaptive navigation framework that leverages learning techniques to enhance exploration efficiency and effectiveness in scalar fields, even under noisy data and obstacles. The framework employs Partial Differential Equations to model scalar fields and a Gaussian Process Regressor to estimate the fields and their gradients, enabling real-time path adjustments and obstacle avoidance. We provide a theoretical foundation for the approach and address several limitations found in existing methods. The effectiveness of our framework is demonstrated through simulation benchmarks and field experiments with an Autonomous Surface Vehicle, showing improved efficiency and adaptability compared to traditional methods and offering a robust solution for real-time environmental monitoring.

Keywords: SLAM, Motion and Path Planning, Autonomous Agents, Foundations of Automation
Abstract: Decision making under uncertainty is at the heart of any autonomous system acting with imperfect information. The cost of solving the decision making problem is exponential in the action and observation spaces, thus rendering it unfeasible for many online systems. This paper introduces a novel approach to efficient decision-making, by partitioning the high-dimensional observation space. Using the partitioned observation space, we formulate analytical bounds on the expected information- theoretic reward, for general belief distributions. These bounds are then used to plan efficiently while keeping performance guar- antees. We show that the bounds are adaptive,

computationally efficient, and that they converge to the original solution. We extend the partitioning paradigm and present a hierarchy of partitioned spaces that allows greater efficiency in planning. We then propose a specific variant of these bounds for Gaussian beliefs and show a theoretical performance improvement of at least a factor of 4. Finally, we compare our novel method to other state of the art algorithms in active SLAM scenarios, in simulation and in real experiments. In both cases we show a sign

Keywords: Deep Learning Methods, Mapping, SLAM
Abstract: Visual Simultaneous Localization and Mapping (SLAM) helps robots estimate their poses and perceive the environment in unknown settings. Recent work has demonstrated that implicit neural radiance fields and 3D Gaussian Splatting (3DGS) offer higher fidelity scene representation than traditional map representations. We propose VSS-SLAM, which utilizes voxelized surfels as the map representation for incremental mapping in unknown environments. This representation effectively addresses the issue of redundant and disordered primitives encountered in previous methods, thereby enhancing geometric accuracy during reconstruction. Specifically, our approach divides the scene using voxels and stores geometric and appearance information in feature vectors at the voxel vertices. Before rendering, these feature vectors are decoded to generate the corresponding surfels. Additionally, we align camera poses through image and depth rendering. Extensive experiments on the Replica and TUM RGBD datasets demonstrate that VSS-SLAM delivers high-fidelity reconstruction and accurate pose estimation in both simulated and real-world environments. Source code will soon be available.

Keywords: Mapping, Autonomous Agents, SLAM
Abstract: Comparing graph-structured maps is a task of paramount importance in robotic exploration and cartography, but unfortunately the computational cost of the existing similarity measures, such as the graph edit distance (GED), is prohibitive for large graphs. In this paper, we introduce and characterize three new graph distance measures which satisfy the requirements for a metric. The first one, "LogEig", computes the square root of the sum of the squared logarithms of the generalized eigenvalues of the shifted Laplacian matrices associated with the two graphs, while the second calculates the Bures distance between these positive definite matrices. The third distance, "Rank", computes the rank of the difference of the graph shift operators associated with the two graphs, e.g. the adjacency or the Laplacian matrix. Examples and numerical experiments with graphs from a publicly-available dataset, show the accuracy and computational efficiency of the new metrics for 2D topological-map matching, compared to the GED. The effect of spectral sparsification on the new graph distance measures is examined as well.

Keywords: Data Sets for SLAM, Data Sets for Robotic Vision, Semantic Scene Understanding
Abstract: To ensure the efficiency of robot autonomy under diverse real-world conditions, a high-quality heterogeneous dataset is essential to benchmark the operating algorithms' performance and robustness. Current benchmarks predominantly focus on urban terrains, specifically for on-road autonomous driving, leaving multi-degraded, densely vegetated, dynamic and feature-sparse environments, such as underground tunnels, natural fields, and modern indoor spaces underrepresented. To fill this gap, we introduce EnvoDat, a large-scale, multi-modal dataset collected in diverse environments and conditions, including high illumination, fog, rain, and zero visibility at different times of the day. Overall, EnvoDat contains 26 sequences from 13 scenes, 10 sensing modalities, over 1.9TB of data, and over 89K fine-grained polygon-based annotations for more than 82 object and terrain classes. We post-processed EnvoDat in different formats that support benchmarking SLAM and supervised learning algorithms, and fine-tuning multimodal vision models. With EnvoDat, we contribute to environment-resilient robotic autonomy in areas where the conditions are extremely challenging. The datasets and other relevant resources can be accessed through https://linusnep.github.io/EnvoDat/.

Keywords: SLAM, Probability and Statistical Methods
Abstract: Degeneracies arising from uninformative geometry are known to deteriorate LiDAR-based localization and mapping. This work introduces a new probabilistic method to detect and mitigate the effect of degeneracies in point-to-plane error minimization. The noise on the Hessian of the point-to-plane optimization problem is characterized by the noise on points and surface normals used in its construction. We exploit this characterization to quantify the probability of a direction being degenerate. The degeneracy-detection procedure is used in a new real-time degeneracy-aware iterative closest point algorithm for LiDAR registration, in which we smoothly attenuate updates in degenerate directions. The method's parameters are selected based on the noise characteristics provided in the LiDAR's datasheet. We validate the approach in four real-world experiments, demonstrating that it outperforms state-of-the-art methods at detecting and mitigating the adverse effects of degeneracies. For the benefit of the community, we release the code for the method at: github.com/ntnu-arl/drpm.

Keywords: SLAM, Mapping, Autonomous Vehicle Navigation
Abstract: In lidar-based Simultaneous Localization and Mapping (SLAM) systems, loop closure detection is crucial for enhancing the accuracy of odometry. However, constraints from loop closure detection are only provided when a loop is detected and can only enhance odometry accuracy at specific moments. Therefore, this paper proposes a lidar inertial odometry system that

periodically provides submap constraints to the pose graph and enhances odometry accuracy through pose graph optimization. In the process of creating submap constraints, the system represents lidar keyframes as a collection of submaps containing overlapping information. The optimal pose transformations between submaps, determined using the Iterative Closest Point (ICP) algorithm with point-to-line and point-to-plane methods, are recognized as submap constraints. During the backend optimization phase, submap constraints and adjacent lidar keyframe constraints are integrated into the pose graph.

The pose graph is then optimized using the pose graph optimization method to achieve the optimal lidar pose estimation. Additionally, To further enhance pose estimation, point-to-plane correspondence is established by considering the differences in normal vectors of feature points between the scan and the map, and integrated initial positioning module is created by incorporating preintegration and scan-to-scan matching. The results of simulation, public datasets and vehicle experiments show that the accuracy of the proposed algorithm is significantly improved compared to the advanced SLAM algorithm.

Keywords: AI-Based Methods, View Planning for SLAM, Motion and Path Planning
Abstract: In this paper, we introduce HDPlanner, a deep reinforcement learning (DRL) based framework designed to tackle two core and challenging tasks

for mobile robots: autonomous exploration and navigation, where the robot must optimize its trajectory adaptively to achieve the task objective through continuous interactions in unknown environments. Specifically, HDPlanner relies on novel hierarchical attention networks

to empower the robot to reason about its belief across multiple spatial

scales and sequence collaborative decisions, where our networks decompose long-term objectives into short-term informative task assignments and informative path plannings. We further propose a contrastive learning-based joint optimization to enhance the robustness

of HDPlanner. We empirically demonstrate that HDPlanner significantly outperforms state-of-the-art conventional and learning-based baselines on an extensive set of simulations, including hundreds of test maps and

large-scale, complex Gazebo environments. Notably, HDPlanner achieves real-time planning with travel distances reduced by up to 35.7% compared to exploration benchmarks and by up to 16.5% than navigation benchmarks. Furthermore, we validate our approach on hardware, where it

generates high-quality, adaptive trajectories in both indoor and outdoor environments, highlighting its real-world applicability without

additional training.

Keywords: Computer Vision for Automation, Computer Vision for Transportation, Visual Learning
Abstract: 3D Gaussian Splatting (3DGS) has gained significant attention for 3D scene reconstruction, but still suffers from complex outdoor environments, especially under adverse weather. This is because 3DGS treats the artifacts caused by adverse weather as part of the scene and will directly reconstruct them, largely reducing the clarity of the reconstructed scene. To address this challenge, we propose WeatherGS, a 3DGS-based framework for reconstructing clear scenes from multi-view images under different weather conditions. Specifically, we explicitly categorize the multi-weather artifacts into the dense particles and lens occlusions that have very different characters, in which the former are caused by snowflakes and rain streaks in the air, and the latter are raised by the precipitation on the camera lens. In light of this, we propose a dense-to-sparse preprocess strategy, which sequentially removes the dense particles by an Atmospheric Effect Filter (AEF) and then extracts the relatively sparse occlusion masks with a Lens Effect Detector (LED). Finally, we train a set of 3D Gaussians by the processed images and generated masks for excluding occluded areas, and accurately recover the underlying clear scene by Gaussian splatting. We conduct a diverse and challenging benchmark to facilitate the evaluation of 3D reconstruction under complex weather scenarios. Extensive experiments on this benchmark demonstrate that our WeatherGS consistently produces high-quality, clean scenes across various weather scenarios, outperforming existing state-of-the-art methods.

Keywords: Computer Vision for Automation, Deep Learning in Grasping and Manipulation, Reinforcement Learning
Abstract: Sim-to-Real refers to the process of transferring policies learned in simulation to the real world, which is crucial for achieving practical robotics applications. However, recent Sim2real methods either rely on a large amount of augmented data or large learning models, which is inefficient for specific tasks. In recent years, with the emergence of radiance field reconstruction methods, especially 3D Gaussian splatting, it has become possible to construct realistic real-world scenes. To this end, we propose RL-GSBridge, a novel real-to-sim-to-real framework which incorporates 3D Gaussian Splatting into the conventional RL simulation pipeline, enabling zero-shot simto-real transfer for vision-based deep reinforcement learning. We introduce a mesh-based 3D GS method with soft binding constraints, enhancing the rendering quality of mesh models. Then utilizing a GS editing approach to synchronize the rendering with the physics simulator, RL-GSBridge could reflect the visual interactions of the physical robot accurately. Through a series of sim-to-real experiments, including grasping and pickand-place tasks, we demonstrate that RL-GSBridge maintains a satisfactory success rate in real-world task completion during sim-to-real transfer. Furthermore, a series of rendering metrics and visualization results indicate that our proposed mesh-based 3D GS reduces artifacts in unstructured objects, demonstrating more realistic rendering performance.

Keywords: Computer Vision for Automation
Abstract: In this paper, we address the critical bottleneck in robotics caused by the scarcity of diverse 3D data by presenting a novel two-stage approach for generating high-quality 3D models from a single image. This method is motivated by the need to efficiently expand 3D asset creation, particularly for robotics datasets, where the variety of object types is currently limited compared to general image datasets. Unlike previous methods that primarily rely on general diffusion priors, which often struggle to align with the reference image, our approach leverages subject-specific prior knowledge. By incorporating subject-specific priors in both geometry and texture, we ensure precise alignment between the generated 3D content and the reference object. Specifically, we introduce a shading mode-aware prior into the NeRF optimization process, enhancing the geometry and refining texture in the coarse outputs to achieve superior quality. Extensive experiments demonstrate that our method significantly outperforms prior approaches. Our approach is well-suited for applications such as novel view synthesis, text-to-3D, and image-to-3D, particularly in the robotics field where diverse 3D data is essential.

Keywords: Computer Vision for Transportation
Abstract: 小说视图合成 （NVS） 方法起着关键作用在大型场景重建中。然而，这些方法依赖于严重依赖密集的图像输入和延长的训练时间，使它们不适合计算资源所在的位置 有限。 此外，小样本方法通常难以在广阔的环境中重建质量差。 这 论文介绍了 DGTR，这是一种新颖的分布式框架，用于稀疏视图 vast 的高效高斯重建 场景。 我们的方法将场景划分为多个区域，由具有稀疏图像输入的无人机独立处理。使用前馈高斯模型，我们预测高质量的高斯基元，然后是全局对齐算法来确保几何一致性。综合视图和深度先验被合并到进一步增强训练，而基于蒸馏的模型聚合机制可实现高效的&#

Keywords: Computer Vision for Transportation, Data Sets for Robotic Vision, Deep Learning for Visual Perception
Abstract: We present LiDAR-EDIT, a novel paradigm for generating synthetic LiDAR data for autonomous driving. Our framework edits real-world LiDAR scans by introducing new object layouts while preserving the realism of the background environment. Compared to end-to-end frameworks that generate LiDAR point clouds from scratch, LiDAR-EDIT offers users full control over the object layout, including the number, type, and pose of objects, while keeping most of the original real-world background. Our method also provides object labels for the generated data. Compared to novel view synthesis techniques, our framework allows for the creation of counterfactual scenarios with object layouts significantly different from the original real-world scene. LiDAR-EDIT uses spherical voxelization to enforce correct LiDAR projective geometry in the generated point clouds by construction. During object removal and insertion, generative models are employed to fill the unseen background and object parts that were occluded in the original real Lidar scans. Experimental results demonstrate that our framework produces realistic LiDAR scans with practical value for downstream tasks. Project website with open-sourced code: https://sites.google.com/view/lidar-edit

Keywords: Mapping, Deep Learning for Visual Perception, Visual Learning
Abstract: High-Definition (HD) map construction is essential for autonomous driving to accurately understand the surrounding environment. Most existing methods rely on single-frame inputs to predict local map, which often fail to effectively capture the temporal correlations between frames. This limitation results in discontinuities and instability in the generated map.To tackle this limitation, we propose a textit{Ti}ghtly textit{C}oupled temporal fusion textit{Map} textit{Net}work (TICMapNet). TICMapNet breaks down the fusion process into three sub-problems: PV feature alignment, BEV feature adjustment, and Query feature fusion. By doing so, we effectively integrate temporal information at different stages through three plug-and-play modules, using the proposed tightly coupled strategy. Unlike traditional methods, our approach does not rely on camera extrinsic parameters, offering a new perspective for addressing the visual fusion challenge in the field of object detection. Experimental results show that TICMapNet significantly improves upon its single-frame baseline model, achieving at least a 7.0% increase in mAP using just two consecutive frames on the nuScenes dataset, while also showing generalizability across other tasks. The code and demos are available at url{https://github.com/adasfag/TICMapNet}.

Keywords: Localization, Mapping
Abstract: The accurate reconstruction of dynamic scenes with neural radiance fields is significantly dependent on the estimation of camera poses. Widely used structure-from-motion pipelines encounter difficulties in accurately tracking the camera trajectory when faced with separate dynamics of the scene content and the camera movement. To address this challenge, we propose Dynamic Motion-Aware Fast and Robust Camera Localization for Dynamic Neural Radiance Fields (DynaMoN). DynaMoN utilizes semantic segmentation and generic motion masks to handle dynamic content for initial camera pose estimation and statics-focused ray sampling for fast and accurate novel-view synthesis. Our novel iterative learning scheme switches between training the NeRF and updating the pose parameters for an improved reconstruction and trajectory estimation quality. The proposed pipeline shows significant acceleration of the training process. We extensively evaluate our approach on two real-world dynamic datasets, the TUM RGB-D dataset and the BONN RGB-D Dynamic dataset. DynaMoN improves over the state-of-the-art both in terms of reconstruction quality and trajectory accuracy. We plan to make our code public to enhance research in this area.

Keywords: Force and Tactile Sensing
Abstract: Vision-based tactile sensors (VBTSs) provide high-resolution tactile images crucial for robot in-hand manipulation. However, force sensing in VBTSs is underutilized due to the costly and time-intensive process of acquiring paired tactile images and force labels. In this study, we introduce a transferable force prediction model, TransForce, designed to leverage collected image-force paired data for new sensors under varying illumination colors and marker patterns while improving the accuracy of predicted forces, especially in the shear direction. Our model effectively achieves translation of tactile images from the source domain to the target domain, ensuring that the generated tactile images reflect the illumination colors and marker patterns of the new sensors while accurately aligning the elastomer deformation observed in existing sensors, which is beneficial to force prediction of new sensors. As such, a recurrent force prediction model trained with generated sequential tactile images and existing force labels is employed to estimate higher-accuracy forces for new sensors with lowest average errors of 0.69N (5.8% in full work range) in x-axis, 0.70N (5.8%) in y-axis, and 1.11N (6.9%) in z-axis compared with models trained with single images. The experimental results also reveal that pure marker modality is more helpful than the RGB modality in improving the accuracy of force in the shear direction, while the RGB modality show better performance in the normal direction.

Keywords: Force and Tactile Sensing, Multi-Modal Perception for HRI, Soft Sensors and Actuators
Abstract: Tactile sensors play a crucial role in enabling robots to interact effectively and safely with objects in everyday tasks. In particular, visuotactile sensors have seen increasing usage in two and three-fingered grippers due to their high-quality feedback. However, a significant gap remains in the development of sensors suitable for humanoid robots, especially five-fingered dexterous hands. One reason is because of the challenges in designing and manufacturing sensors that are compact in size. In this paper, we propose HumanFT, a multimodal visuotactile sensor that replicates the shape and functionality of a human fingertip. To bridge the gap between human and robotic tactile sensing, our sensor features real-time force measurements, high-frequency vibration detection, and overtemperature alerts. To achieve this, we developed a suite of fabrication techniques for a new type of elastomer optimized for force propagation and temperature sensing. Besides, our sensor integrates circuits capable of sensing pressure and vibration. These capabilities have been validated through experiments. The proposed design is simple and cost-effective to fabricate. We believe HumanFT can enhance humanoid robots' perception by capturing and interpreting multimodal tactile information.

Keywords: Force and Tactile Sensing, Perception for Grasping and Manipulation
Abstract: In this paper, we tackle the problem of estimating 3D contact forces using vision-based tactile sensors. In particular, our goal is to estimate contact forces over a large range (up to 15 N) on any objects while generalizing across different vision-based tactile sensors. Thus, we collected a dataset of over 200K indentations using a robotic arm that pressed various indenters onto a GelSight Mini sensor mounted on a force sensor and then used the data to train a multi-head transformer for force regression. Strong generalization is achieved via accurate data collection and multi-objective optimization that leverages depth contact images. Despite being trained only on primitive shapes and textures, the regressor achieves a mean absolute error of 4% on a dataset of unseen real-world objects. We further evaluate our approach's generalization capability to other GelSight mini and DIGIT sensors, and propose a reproducible calibration procedure for adapting the pre-trained model to other vision-based sensors. Furthermore, the method was evaluated on real-world tasks, including weighing objects and controlling the deformation of delicate objects, which relies on accurate force feedback.

Keywords: Force and Tactile Sensing, Deep Learning in Grasping and Manipulation, Imitation Learning
Abstract: Tactile information is a critical tool for dexterous manipulation. As humans, we rely heavily on tactile information to understand objects in our environments and how to interact with them. We use touch not only to perform manipulation tasks but also to learn how to perform these tasks. Therefore, to create robotic agents that can learn to complete manipulation tasks at a human or super-human level of performance, we need to properly incorporate tactile information into both skill execution and skill learning. In this paper, we investigate how we can incorporate tactile information into imitation learning platforms to improve performance on manipulation tasks. We show that incorporating visuo-tactile pretraining improves imitation learning performance, not only for tactile agents (policies that use tactile information at inference), but also for non-tactile agents (policies that do not use tactile information at inference). For these non-tactile agents, pretraining with tactile information significantly improved performance (for example, improving the accuracy on USB plugging from 20% to 85%), reaching a level on par with visuo-tactile agents, and even surpassing them in some cases. For demonstration videos and access to our codebase, see the project website: https://sites.google.com/andrew.cmu.edu/visuo-tactile-pretr aining

Keywords: Force and Tactile Sensing, Contact Modeling, Haptics and Haptic Interfaces
Abstract: For elastomer-based tactile sensors, represented by vision-based tactile sensors, routine calibration of mechanical parameters (Young's modulus and Poisson's ratio) has been shown to be important for force reconstruction. However, the reliance on existing in-situ calibration methods for accurate force measurements limits their cost-effective and flexible applications. This article proposes a new in-situ calibration scheme that relies only on comparing contact deformation. Based on the detailed derivations of the normal contact and torsional contact theories, we designed a simple and low-cost calibration device, EasyCalib, and validated its effectiveness through extensive finite element analysis. We also explored the accuracy of EasyCalib in the practical application and demonstrated that accurate contact distributed force reconstruction can be realized based on the mechanical parameters obtained. EasyCalib balances low hardware cost, ease of operation, and low dependence on technical expertise and is expected to provide the necessary accuracy guarantees for wide applications of visuotactile sensors.

Keywords: Force and Tactile Sensing, Perception for Grasping and Manipulation
Abstract: Tactile sensing is crucial for robots aiming to achieve human-level dexterity. Among tactile-dependent skills, tactile-based object tracking serves as the cornerstone for many tasks, including manipulation, in-hand manipulation, and 3D reconstruction. In this work, we introduce NormalFlow, a fast, robust, and real-time tactile-based 6DoF tracking algorithm. Leveraging the precise surface normal estimation of vision-based tactile sensors, NormalFlow determines object movements by minimizing discrepancies between the tactile-derived surface normals. Our results show that NormalFlow consistently outperforms competitive baselines and can track low-texture objects like table surfaces. For long-horizon tracking, we demonstrate when rolling the sensor around a bead for 360 degrees, NormalFlow maintains a rotational tracking error of 2.5 degrees. Additionally, we present state-of-the-art tactile-based 3D reconstruction results, showcasing the high accuracy of NormalFlow. We believe NormalFlow unlocks new possibilities for high-precision perception and manipulation tasks that involve interacting with objects using hands. Please also check our supplementary video to see our method in action.

Keywords: Marine Robotics, Field Robots, Aerial Systems: Applications
Abstract: 航空水下航行器 （HAUV） 由于能够在空中和水域中无缝运行，因此表现出巨大的潜力。然而，在两种介质的快速可作性和在跨域阶段实现稳定性仍然是一个重大挑战。受到可伸缩四肢的启发，本文提出了一种新颖的变形 HAUV，哪吒-MB。哪吒-MB 在过渡阶段利用线性执行器结合齿轮和齿条系统进行手臂转换，取代传统的伺服系统。转化机构占总重量的 11%。在空中模式下，哪吒-MB 表现出与四旋翼配置相当的飞行性能。在水下模式下，哪吒-MB 将其四旋翼臂缩回子弹形外壳中，显着减少阻力和能耗，同时能够通过直径小至 134 毫米的狭窄间隙。在空中和水下领域进行的模拟和现场测试表ą

Keywords: Aerial Systems: Applications, Surveillance Robotic Systems, AI-Enabled Robotics
Abstract: This work demonstrates universal dynamic perching capabilities for quadrotors of various sizes and on surfaces with different orientations. By employing a non-dimensionalization framework and deep reinforcement learning, we systematically assessed how robot size and surface orientation affect landing capabilities. We hypothesized that maintaining geometric proportions across different robot scales ensures consistent perching behavior, which was validated in both simulation and experimental tests. Additionally, we investigated the effects of joint stiffness and damping in the landing gear on perching behaviors and performance. While joint stiffness had minimal impact, joint damping ratios influenced landing success under vertical approaching conditions. The study also identified a critical velocity threshold necessary for successful perching, determined by the robot's maneuverability and leg geometry. Overall, this research advances robotic perching capabilities, offering insights into the role of mechanical design and scaling effects, and lays the groundwork for future drone autonomy and operational efficiency in unstructured environments.

Keywords: Field Robots, Aerial Systems: Applications
Abstract: This work deals with the problem of unlocking perpetual deployment capabilities for small-UAS robotics across the diverse settings of the real world and their challenges, encompassing considerations for marine environments alongside the more common terrestrial ones. Via the progress made within this scope, a step towards truly ubiquitous and self-sustainable aerial robotics is accomplished. The work consists of the development of the Gannet Solar-VTOL, a waterproof small-UAS that is capable of resting on the surface of water for prolonged periods of time and over varying temperature ranges, while harvesting solar power to recharge itself. Equally importantly, it integrates a field-proven Self-Sustainable Autonomous System architecture that allows it to hibernate and sustain its battery charge overnight or during periods of solar illumination scarcity, as well as to assess mission-critical parameters (e.g., water surface turbulence, ambient temperature of battery compartment) on the low-power side of the Power Management Stack, and react appropriately. Finally, the robot is equipped with an onboard camera and a Neural Processing Unit that allows it to perform in-field environmental monitoring operations (e.g., wildfire detection). This paper experimentally demonstrates the aforementioned capabilities, and concludes with a presentation of the amphibious small-UAS' long-term deployment within a marine environment in the N.Nevada region, spanning over 3 consecutive days.

Keywords: Aerial Systems: Perception and Autonomy, Reinforcement Learning, Vision-Based Navigation
Abstract: This paper presents a novel autonomous drone-based smoke plume tracking system capable of navigating and tracking plumes in highly unsteady atmospheric conditions. The system integrates advanced hardware and software and a comprehensive simulation environment to ensure robust performance in controlled and real-world settings. The quadrotor, equipped with a high-resolution imaging system and an advanced onboard computing unit, performs precise maneuvers while accurately detecting and tracking dynamic smoke plumes under fluctuating conditions. Our software implements a two-phase flight operation: descending into the smoke plume upon detection and continuously monitoring the smoke's movement during in-plume tracking. Leveraging Proportional Integral–Derivative (PID) control and a Proximal Policy Optimization (PPO) based Deep Reinforcement Learning (DRL) controller enables adaptation to plume dynamics. Unreal Engine simulation evaluates performance under various smoke-wind scenarios, from steady flow to complex, unsteady fluctuations, showing that while the PID controller performs adequately in simpler scenarios, the DRL-based controller excels in more challenging environments. Field tests corroborate these findings. This system opens new possibilities for drone-based monitoring in areas like wildfire management and air quality assessment. The successful integration of DRL for real-time decision-making advances autonomous drone control for dynamic environments.

Keywords: Mapping, Field Robots, Aerial Systems: Applications
Abstract: Traditionally, unmanned aerial vehicles (UAVs) rely on CMOS-based cameras to collect images about the world below. One of the most successful applications of UAVs is to generate orthomosaics or orthomaps, in which a series of images are integrated to develop a larger map. However, using CMOS-based cameras with global or rolling shutters means that orthomaps are vulnerable to challenging light conditions, motion blur, and high-speed motion of independently moving objects (IMOs) under the camera. Event cameras are less sensitive to these issues, as their pixels trigger asynchronously on brightness changes. This work introduces the first orthomosaic approach using event cameras. We focus on addressing high-dynamic range and low-light problems in orthomosaics. In contrast to existing methods relying only on CMOS cameras, our approach enables map generation even in challenging light conditions, including direct sunlight and after sunset.

Keywords: Aerial Systems: Perception and Autonomy, Simulation and Animation, Software, Middleware and Programming Environments
Abstract: Uncrewed Aerial Vehicle (UAV) research faces challenges with safety, scalability, costs, and ecological impact when conducting hardware testing. High-fidelity simulators offer a vital solution by replicating real-world conditions to enable the development and evaluation of novel perception and control algorithms. However, the large number of available simulators poses a significant challenge for researchers to determine which simulator best suits their specific use-case, based on each simulator’s limitations and customization readiness. In this paper we present an overview of 44 UAV simulators, including in-depth, systematic comparisons for 14 of the simulators. Additionally, we present a set of decision factors for selection of simulators, aiming to enhance the efficiency and safety of research endeavors.

Keywords: Aerial Systems: Applications, Calibration and Identification, Robust/Adaptive Control
Abstract: The widespread adoption of quadrotors for diverse applications, from agriculture to public safety, necessitates an understanding of the aerodynamic disturbances they create. This paper introduces a computationally lightweight model for estimating the time-averaged magnitude of the induced flow below quadrotors in hover. Unlike related

approaches that rely on expensive computational fluid dynamics (CFD) simulations or drone specific time-consuming empirical measurements, our method leverages classical theory from turbulent flows. By analyzing over 16 hours of flight data from drones of varying sizes within a large motion capture system, we show for the first time that the combined flow from all drone propellers is well-approximated by a turbulent jet after 2.5 drone-diameters below the vehicle. Using a novel normalization and scaling, we experimentally identify model parameters that describe a unified mean velocity field below differently sized quadrotors. The model, which requires only the drone's mass, propeller size, and drone size for calculations, accurately describes the far-field airflow over a long-range in a very large volume which is impractible to simulate using CFD. Our model offers a practical tool for ensuring safer operations near humans, optimizing sensor placements and drone control in multi-agent scenarios. We demonstrate the latter by designing a controller that compensates for the downwash of another drone, leading to a four times lower altitude deviation when passing below.

Keywords: Object Detection, Segmentation and Categorization, Multi-Robot Systems, Intelligent Transportation Systems
Abstract: Collaborative perception, fusing information from multiple agents, can extend perception range so as to improve perception performance. However, temporal asynchrony in real-world environments, caused by communication delays, clock misalignment, or sampling configuration differences, can lead to information mismatches. If this is not well handled, then the collaborative performance is patchy, and what's worse safety accidents may occur. To tackle this challenge, we propose CoDynTrust, an uncertainty-encoded asynchronous fusion perception framework that is robust to the information mismatches caused by temporal asynchrony. CoDynTrust generates dynamic feature trust modulus (DFTM) for each region of interest by modeling aleatoric and epistemic uncertainty as well as selectively suppressing or retaining single-vehicle features, thereby mitigating information mismatches. We then design a multi-scale fusion module to handle multi-scale feature maps processed by DFTM. Compared to existing works that also consider asynchronous collaborative perception, CoDynTrust combats various low-quality information in temporally asynchronous scenarios and allows uncertainty to be propagated to downstream tasks such as planning and control. Experimental results demonstrate that CoDynTrust significantly reduces performance degradation caused by temporal asynchrony across multiple datasets, achieving state-of-the-art detection performance even with temporal asynchrony. The code is available at https://github.com/CrazyShout/CoDynTrust.

Keywords: Object Detection, Segmentation and Categorization, Deep Learning for Visual Perception, Recognition
Abstract: This paper tackles the challenging problem of semi-supervised monocular 3D object detection with a general framework. In specific, having observed that the bottleneck of this task lies in lacking reliable and informative samples from unlabeled data for detector learning, we introduce a novel simple yet effective `Augment and Criticize' pipeline that mines abundant informative samples for robust detection. To be more specific, in the `Augment' stage, we present the Augmentation-based Prediction aGgregation (APG), which applies automatically learned transformations to unlabeled images and aggregates detections from various augmented views as pseudo labels. Since not all the pseudo labels from APG are beneficially informative, the subsequent `Criticize' phase is introduced. Particularly, we present the Critical Retraining Strategy (CRS) that, unlike simply filtering pseudo labels using a fixed threshold, employs a learnable network to evaluate the contribution of unlabeled images at different training timestamps. This way, the noisy samples prohibitive to model evolution can be effectively suppressed. In order to validate `Augment-Criticize', we apply it to MonoDLE and MonoFlex, and the two new detectors, dubbed 3DSeMo_DLE and 3DSeMo_FLEX, achieve state-of-the-art results with consistent improvements, evidencing its effectiveness and generality.

Keywords: Object Detection, Segmentation and Categorization, Computer Vision for Transportation, Deep Learning for Visual Perception
Abstract: We tackle the problem of monocular 3D object detection across different sensors, environments, and camera setups. In this paper, we introduce a novel unsupervised domain adaptation approach, MonoCT, that generates highly accurate pseudo labels for self-supervision. Inspired by our observation that accurate depth estimation is critical to mitigating domain shifts, MonoCT introduces a novel Generalized Depth Enhance- ment (GDE) module with an ensemble concept to improve depth estimation accuracy. Moreover, we introduce a novel Pseudo Label Scoring (PLS) module by exploring inner-model consistency measurement and a Diversity Maximization (DM) strategy to further generate high-quality pseudo labels for self-training. Extensive experiments on six benchmarks show that MonoCT outperforms existing SOTA domain adaptation methods by large margins (∼21% minimum for AP Mod.) and generalizes well to car, traffic camera and drone views.

Keywords: AI-Based Methods, Localization, SLAM
Abstract: SLAM is an important capability for many autonomous systems, and modern LiDAR-based methods offer promising performance. However, for long duration missions, existing works that either take directly the full pointclouds or extracted features face key tradeoffs in accuracy and computational efficiency (e.g., memory consumption). To address these issues, this paper presents DFLIOM with several key innovations. Unlike previous methods that rely on handcrafted heuristics and hand-tuned parameters for feature extraction, we propose a learning-based approach that select points relevant to LiDAR SLAM pointcloud registration. Furthermore, we extend our prior work DLIOM with the learned feature extractor and observe our method enables similar or even better localization performance using only about 20% of the points in the dense point clouds. We demonstrate that DFLIOM performs well on multiple public benchmarks, achieving a 2.4% decrease in localization error and 57.5% decrease in memory usage compared to state-of-the-art methods (DLIOM). Although extracting features with the proposed network requires extra time, it is offset by the faster processing time downstream, thus maintaining real-time performance using 20Hz LiDAR on our hardware setup. The effectiveness of our learning-based feature extraction module is further demonstrated through comparison with several handcrafted feature extractors.

Keywords: Computer Vision for Automation, Autonomous Vehicle Navigation, Virtual Reality and Interfaces
Abstract: Synthesizing photo-realistic visual observations from an ego vehicle's driving trajectory is a critical step towards scalable training of self-driving models. Reconstruction-based methods create 3D scenes from driving logs and synthesize geometry-consistent driving videos through neural rendering, but their dependence on costly object annotations limits their ability to generalize to in-the-wild driving scenarios. On the other hand, generative models can synthesize action-conditioned driving videos in a more generalizable way but often struggle with maintaining 3D visual consistency. In this paper, we present ourmethod{}, a 4D spatial-temporal scene generation approach that combines the merits of generation and reconstruction, to synthesize generalizable 4D driving scenes and dynamic driving videos with 3D consistency. Specifically, we leverage the generative power of video diffusion models to synthesize a sequence of visual references and further elevate them to 4D with a novel hybrid Gaussian representation. Given a driving trajectory, we then render 3D-consistent driving videos via Gaussian splatting. The use of generative priors allows our method to produce high-quality 4D scenes from in-the-wild driving data, while neural rendering ensures 3D-consistent video generation from the 4D scenes. Extensive experiments on nuScenes and in-the-wild driving data demonstrate that ourmethod{} can generate controllable and generalizable 4D driving scenes, synthesize novel views of driving videos with high fidelity and 3D consistency, decompose static and dynamic elements in a self-supervised manner, and enhance perception and planning tasks for autonomous driving.

Keywords: Visual Learning, Incremental Learning, Mapping
Abstract: We present a framework, DISORF, to enable online 3D reconstruction and visualization of scenes captured by resource-constrained mobile robots and edge devices. To address the limited computing capabilities of edge

devices and potentially limited network availability, we design a framework that efficiently distributes computation between the edge device and the remote server. We leverage on-device SLAM systems to generate posed keyframes and transmit them to remote servers that can perform high-quality 3D reconstruction and visualization at runtime by leveraging recent advances in neural 3D methods. We identify a key challenge with online training where naive image sampling strategies can lead to significant degradation in rendering quality. We propose a novel shifted exponential frame sampling method that addresses this challenge for online training. We demonstrate the effectiveness of our framework in enabling high-quality real-time reconstruction and visualization of unknown scenes as they are captured and streamed from cameras in mobile robots and edge devices.

Keywords: Reactive and Sensor-Based Planning, Integrated Planning and Control, Motion and Path Planning
Abstract: Safe autonomous navigation in a priori unknown environments is an essential skill for mobile robots to reliably and adaptively perform diverse tasks (e.g., delivery, inspection, and interaction) in unstructured cluttered environments. Hybrid metric-topological maps, constructed as a pose graph of local submaps, offer a computationally efficient world representation for adaptive mapping, planning, and control at the regional level. In this paper, we consider a pose graph of locally sensed star-convex scan regions as a metric-topological map, with star convexity enabling simple yet effective local navigation strategies. We design a new family of safe local scan navigation policies and present a perception-driven feedback motion planning method through the sequential composition of local scan navigation policies, enabling provably correct and safe robot navigation over the union of local scan regions. We introduce a new concept of frontier and bridging scans for automated key scan selection and exploration for integrated mapping and navigation in unknown environments. We demonstrate the effectiveness of our key-scan-based navigation and mapping framework using a mobile robot equipped with a 360˝ laser range scanner in 2D cluttered environments through numerical ROS-Gazebo simulations and real hardware experiments.

Keywords: Humanoid and Bipedal Locomotion, Body Balancing, Whole-Body Motion Planning and Control
Abstract: In this work, the Divergent Component of Motion (DCM) method is expanded to include angular coordinates for the first time. This work introduces the idea of spatial DCM, which adds an angular objective to the existing linear DCM theory. To incorporate the angular component into the framework, a discussion is provided on extending beyond the linear motion of the Linear Inverted Pendulum model (LIPM) towards the Single Rigid Body model (SRBM) for DCM. This work presents the angular DCM theory for a 1D rotation, simplifying the SRBM rotational dynamics to a flywheel to satisfy necessary linearity constraints. The 1D angular DCM is mathematically identical to the linear DCM and defined as an angle which is ahead of the current body rotation based on the angular velocity. This theory is combined into a 3D linear and 1D angular DCM framework, with discussion on the feasibility of simultaneously achieving both sets of objectives. A simulation in MATLAB and hardware results on the TORO humanoid are presented to validate the framework's performance.

Keywords: Humanoid and Bipedal Locomotion, Legged Robots, Humanoid Robot Systems
Abstract: This paper presents a Model Predictive Control (MPC) formulation for bipedal footstep planning based on the Linear Inverted Pendulum (LIP) model, ensuring recursive feasibility when navigating restricted regions. The proposed approach incorporates capturability and introduces a new constraint that forces the Divergent Component of Motion (DCM) into a finite-step capture region, adjusted between consecutive MPC calls. This constraint enables the MPC to anticipate beyond its prediction horizon, preventing collisions with the walking surface boundaries. We validate the approach through high-fidelity simulations with the bipedal robot Digit, demonstrating recursively feasible MPC footstep planning in restricted regions. Future efforts will extend the approach to general polytopic constraints, thereby facilitating footstep planning in cluttered environments while preserving the MPC's recursive feasibility.

Keywords: Humanoid and Bipedal Locomotion, Whole-Body Motion Planning and Control, Optimization and Optimal Control
Abstract: One of the essential aspects of humanoid robot running is determining the limb-swinging trajectories. During the flight phases, where the ground reaction forces are not available for regulation, the limb swinging trajectories are significant for the stability of the next stance phase. Due to the conservation of angular momentum, improper leg and arm swinging results in highly tilted and unsustainable body configurations at the next stance phase landing. In such cases, the robotic system fails to maintain locomotion independent of the stability of the center of mass trajectories. This problem is more apparent for fast and high flight time trajectories. This paper proposes a real-time nonlinear limb trajectory optimization problem for humanoid running. The optimization problem is tested on two different humanoid robot models, and the generated trajectories are verified using a running algorithm for both robots in a simulation environment.

Keywords: Modeling and Simulating Humans, Humanoid Robot Systems, Human and Humanoid Motion Analysis and Synthesis
Abstract: Humanoid robots that mimic human movement have garnered significant attention in recent years. This study focuses on mimicking the efficient pitching motion of humans by incorporating two main approaches into a humanoid robot: (1) the use of elastic elements to assist joint torque, and (2) the optimization of motor torque to minimize energy consumption. This robot is intended to emulate human physical characteristics, such as mass, link length, and center of gravity, with a particular focus on utilizing the elastic energy generated during shoulder internal and external rotation. A leaf spring is attached in parallel with the motor at the shoulder pitch joint to release the elastic energy stored during shoulder external rotation, thereby assisting internal rotation in a manner similar to human biomechanics. Additionally, motor torque optimization is performed using Fujitsu's Digital Annealer to generate energy-efficient motions. Experiments conducted through simulations and with an actual pitching robot assessed the effectiveness of these technologies in mimicking human-like pitching motion. The results suggest that combining elastic elements with motion optimization techniques enable robots to achieve more efficient human-like movements.

Keywords: Legged Robots, Optimization and Optimal Control, Passive Walking
Abstract: Open-loop stable limit cycles are foundational to legged robotics, providing inherent self-stabilization that minimizes the need for computationally intensive feedback-based gait correction. While previous methods have primarily targeted specific robotic models, this paper introduces a general framework for rapidly generating limit cycles across various dynamical systems, with the flexibility to impose arbitrarily tight stability bounds. We formulate the problem as a single-stage constrained optimization problem and use Direct Collocation to transcribe it into a nonlinear program with closed-form expressions for constraints, objectives, and their gradients.

Our method supports multiple stability formulations. In particular, we tested two popular formulations for limit cycle stability in robotics: (1) based on the spectral radius of a discrete return map, and (2) based on the spectral radius of the monodromy matrix, and tested five different constraint-satisfaction formulations of the eigenvalue problem to bound the spectral radius. We compare the performance and solution quality of the various formulations on a robotic swing-leg model, highlighting the Schur decomposition of the monodromy matrix as a method with broader applicability due to weaker assumptions and stronger numerical convergence properties.

As a case study, we apply our method on a hopping robot model, generating open-loop stable gaits in under 2 seconds on an Intel Core i7-6700K, while simultaneously minimizing energy consumption even under tight stability constraints.

Keywords: Legged Robots, Dynamics, Humanoid and Bipedal Locomotion
Abstract: This paper presents a periodic running control strategy based on a modified Spring-Loaded Inverted Pendulum (SLIP) model to achieve stable

running at various velocities. While the traditional SLIP model is valued for its simplicity and intuitive representation of running dynamics, its limitations impede its extension and integration with feedback control systems. To address this, we introduce a novel Quasi-Linearized SLIP model (QLSLIP) that incorporates additional forces in the radial and angular directions to enable stable running across various velocities. This model simplifies the analytical representation of the stance phase and defines the required swept angle

for maintaining periodic motion during the flight phase. Using this model, we develop a feedback control system that ensures the stability of QLSLIP-based periodic locomotion, even in the presence of external disturbances. This control framework optimizes trajectories and sustains periodic motion in real-time across diverse scenarios. Additionally, we propose an algorithm to extend this approach to articulated leg mechanisms. The effectiveness of the proposed algorithm

is validated through simulations under various conditions, demonstrating improvements in the stability and performance of running.

Keywords: Legged Robots, Dynamics, Compliance and Impedance Control, Granular Media
Abstract: Legged robot locomotion is hindered by a mismatch between applications featuring deformable substrates, where legs can outperform wheels or treads, and existing planners and controllers, most of which assume flat, rigid substrates. In this study we focus on the effects of plastic ground deformation on the hop-to-hop energy dynamics of a hopping robot driven by a switched-compliance energy injection controller. We derive a hop-to-hop energy return map, and we use experiments and simulations to validate this map for a real robot hopping on a real deformable substrate. By analyzing the map’s fixed points and eigenvalues, we identify constant-fixed-point surfaces in parameter space that suggest it is possible to tune control parameters for efficiency or responsiveness while targeting a desired gait energy level. We also identify conditions for which the map’s fixed points are globally stable, and we characterize the basins of attraction of fixed points when these conditions are not satisfied. We conclude by discussing the implications of this energy map for planning, control, and estimation for efficient, agile, and robust legged locomotion on deformable terrain.

Keywords: Surgical Robotics: Steerable Catheters/Needles, Medical Robots and Systems, Deep Learning Methods
Abstract: Tendon-driven continuum robots have been gaining popularity in medical applications due to their ability to curve around complex anatomical structures, potentially reducing the invasiveness of surgery. However, accurate modeling is required to plan and control the movements of these flexible robots. Physics-based models have limitations due to unmodeled effects, leading to mismatches between model prediction and actual robot shape. Recently proposed learning-based methods have been shown to overcome some of these limitations but do not account for hysteresis, a significant source of error for these robots. To overcome these challenges, we propose a novel deep decoder neural network that predicts the complete shape of tendon-driven robots using point clouds as the shape representation, conditioned on prior configurations to account for hysteresis. We evaluate our method on a physical tendon-driven robot and show that our network model accurately predicts the robot's shape, significantly outperforming a state-of-the-art physics-based model and a learning-based model that does not account for hysteresis.

Keywords: Surgical Robotics: Laparoscopy, Semantic Scene Understanding, Medical Robots and Systems
Abstract: Accurate anatomical landmark tracking is crucial yet challenging in laparoscopic surgery due to the changing appearance of landmarks during

dynamic tool-anatomy interactions and visual domain shifts between cases. Unlike appearance-based detection methods, this work proposes a novel graph-based approach to reconstruct the entire target landmark area by explicitly modeling the evolving spatial relations over time among scenario entities, including observable regions, surgical tools, and landmarks. Considering tool-anatomy interactions, we present the Tool-Anatomy Interaction Graph (TAI-G), a spatio-temporal graph that captures spatial dependencies among entities, attribute interactions within entities, and temporal dependencies of spatial relations. To mitigate domain shifts, geometric segmentation features are designated as node attributes, representing domain-invariant image information in the graph space. Message passing with attention helps propagate information across TAI-G, enhancing robust tracking by reconstructing landmark data. Evaluated on laparoscopic cholecystectomy, our framework demonstrates effective handling of complex tool-anatomy interactions and visual domain gaps to accurately track landmarks, showing promise in enhancing the stability and reliability of intricate surgical tasks.

Keywords: Reinforcement Learning, Medical Robots and Systems, Vision-Based Navigation
Abstract: Percutaneous coronary intervention (PCI) involves the insertion of a catheter or guidewire into a blood vessel of a patient, which poses a problem as a doctor is exposed to radiation during the procedure. The use of assistive robots has been proposed to address this issue. Furthermore, recent research is progressing toward complete autonomous navigation using deep reinforcement learning (DRL). Nevertheless, existing algorithms face limitations when operating in numerous unseen environments close to real PCI. This study proposes a robust DRL framework for image-based guidewire navigation to overcome the limitation. We introduce a subtasks strategy and domain randomization to improve robustness in various environments. The subtasks strategy consistently addresses complex global tasks by breaking them into subtasks designed using local maps, allowing them to be robustly solved by a single agent. Domain randomization is applied to handle real PCI issues, including variations in vessel geometry, guidewire deformation, and camera settings. By integrating the two novel methods, our DRL algorithm demonstrates superior performance compared to existing methods across various challenging simulation and phantom environments, validating its effectiveness in real-world scenarios. A video of our experiment is available at url{https://youtu.be/93Q88gESzOY}.

Keywords: Deep Learning Methods, Computer Vision for Medical Robotics
Abstract: 在医学图像分割任务中，扩散模型具有 显示出巨大的潜力。然而，主流 扩散模型显示出包括多次采样在内的缺点 时间和慢速预测结果。最近，作为 独立生成网络，一致性模型具有 解决了现有问题。与扩散模型相比， 一致性模型可以将采样时间缩短到一次， 不仅可以实现类似的生成效果，而且 显著加快训练和预测速度。 但是，它们不适合图像分割 任务。同时，它们在医学成像中的应用 Field 尚未接受调查。因此，本研究 采用一致性模型执行医学影像 分割任务，设计多尺度特征信号 监管模式及损失功能引导实现 模型收敛。实验表明， CTS 模型能够获得更好的医学图像 分Ó

Keywords: Prosthetics and Exoskeletons, Deep Learning Methods, Force and Tactile Sensing
Abstract: Electromyography (EMG) signals are widely used as control inputs for myoelectric exoskeletons. However, muscle fatigue, which can result from prolonged use or heavy loads, significantly affects muscle activation patterns, leading to reduced estimation accuracy. To address this challenge, we propose an adversarial learning framework to enhance grip force estimation under fatigue conditions. The framework consists of three key components: a domain-invariant feature extractor to mitigate domain shifts between non-fatigue and fatigue states, a force estimator to predict grip forces from these domain-invariant features, and a domain discriminator to distinguish between the two domains. The proposed method was evaluated on a dataset collected from eight participants performing gripping tasks under both non-fatigue and fatigue conditions, during which high-density EMG signals and grip forces were recorded simultaneously. Experimental results demonstrated that our method significantly reduced the root mean square error (RMSE) from 0.264 to 0.127, outperforming a baseline model consisting of only the feature extractor and force estimator (p < 0.01). Additionally, the proposed approach exhibited consistent performance across all participants, highlighting its robustness and generalizability. These findings suggest that the proposed adversarial learning framework effectively enhances grip force estimation accuracy under muscle fatigue, offering a promising solution for improving the reliability and usability of myoelectric exoskeletons.

Keywords: Soft Sensors and Actuators, Force and Tactile Sensing, Medical Robots and Systems
Abstract: Video Capsule Endoscopy (VCE) is currently one of the most effective methods for detecting intestinal diseases. However, it is challenging to detect early-stage small nodules with this method because they lack obvious color or shape features. In this letter, we present a new origami capsule endoscope to detect early small intestinal nodules using tactile sensing. Four soft tactile sensors made out of piezoresistive material feed four channels of phase-shifted data that are processed using a particle filter. The particle filter uses an importance assignment template designed using experimental data from six known sizes ofnodules. Moreover, the proposed capsule can use shape changes to passively move forward or backward under peristalsis, enabling it to reach any position in the intestine for detection. Experimental results show that the proposed capsule can detect nodules of more than 3mm diameter with 100% accuracy.

Keywords: Surgical Robotics: Laparoscopy, Medical Robots and Systems, Performance Evaluation and Benchmarking
Abstract: The JHU-ISI Gesture and Skill Assessment Working Set (JIGSAWS) dataset has proven to be a foundational component of modern work on the skill analysis of robotic surgeons. In particular, methods using either the system's kinematics or video data have shown to be able to classify operators into distinct experience levels, and recent approaches have even ventured to recover numeric skill ratings assigned to assessment sessions. Although prior works have achieved positive results in these directions, challenges still remain with classification across all three levels of operator training amounts and objective skill rating regressions. To this end, we perform the first statistical analysis of the dataset itself and compile the results here. We find limited relationships between the amount of experience or training of an operator and their performance in JIGSAWS. Moreover, as operator-side kinematics have well-known relationships with their skill, previous works have used both robot and operator-side kinematics to classify operator skill; we find the first explicit relationships between pure robot-side kinematics and surgical performance. Finally, we analyze the robotic kinematic trends associated with high performance in JIGSAWS tasks and present how they may be used as indicators in human and automated surgeon training.

Keywords: Integrated Planning and Learning, Motion and Path Planning, Autonomous Vehicle Navigation
Abstract: This work presents a novel data-driven path planning algorithm named Instruction-Guided Probabilistic Roadmap (IG-PRM). Despite the recent development and widespread use of mobile robot navigation, the safe and effective travels of mobile robots still require significant engineering effort to take into account the constraints of robots and their tasks. With IG-PRM, we aim to address this problem by allowing robot operators to specify such constraints through natural language instructions, such as ``aim for wider paths'' or ``mind small gaps''. The key idea is to convert such instructions into embedding vectors using large-language models (LLMs) and use the vectors as a condition to predict instruction-guided cost maps from occupancy maps. By constructing a roadmap based on the predicted costs, we can find instruction-guided paths via the standard shortest path search. Experimental results demonstrate the effectiveness of our approach on both synthetic and real-world indoor navigation environments.

Keywords: Motion and Path Planning, Collision Avoidance, Integrated Planning and Control
Abstract: Navigation Among Movable Obstacles (NAMO) poses a challenge for traditional path-planning methods when obstacles block the path, requiring push actions to reach the goal. We propose a framework that enables movability-aware planning to overcome this challenge without relying on explicit obstacle placement. Our framework integrates a global Semantic Visibility Graph and a local Model Predictive Path Integral (SVG-MPPI) approach to efficiently sample rollouts, taking into account the continuous range of obstacle movability. A physics engine is adopted to simulate the interaction result of the rollouts with the environment, and generate trajectories with lower contact force. Qualitative and quantitative experiments suggest that SVG-MPPI outperforms existing paradigm that uses only binary movability for planning, achieving higher success rates with reduced cumulative contact forces. Our code is available at: https://github.com/tud-amr/SVG-MPPI

Keywords: Motion and Path Planning, Collision Avoidance
Abstract: This paper introduces a tangent line decomposition (TLD) algorithm that efficiently finds collision-free paths close to optimal in both 2D and 3D environment. Compared with the existing visibility line-based algorithms, the proposed algorithm innovatively proposed the concept of tangent line decomposition, which decomposes complicated planning into many simple steps. For each step, only one key obstacle is taken into consideration. Besides, instead of constructing a complete graph, a best-first search algorithm is used to avoid searching redundant edges. The path planned by the algorithm is not the optimal path. However, following the idea of the informed RRT* algorithm, the path length planned by TLD can be used as a precondition for other optimal algorithms. In this way, the overall efficiency can be significantly improved. The simulations show that the proposed methods outperform existing methods regarding planning efficiency and solution quality.

Keywords: Motion and Path Planning, Aerial Systems: Applications, Aerial Systems: Perception and Autonomy
Abstract: This paper presents a three-dimensional discrete search path planner for fixed-wing aircraft emergency landing planning that manages state-space complexity by incorporating cost gradients to assure descent flight path angle and runway heading alignment constraints are met. Our approach incorporates steady wind and maximizes margin from flight envelope boundaries to accommodate wind variation in a manner commensurate with a loss of thrust condition. A novel multi-objective cost function that combines gradient-based path guidance and population risk metrics is implemented to efficiently enable discrete search to find a robust solution. The proposed method is demonstrated through use cases with population data for a region of Long Island, New York that highlight our algorithm's effectiveness.

Keywords: Motion and Path Planning
Abstract: This paper investigates Path planning Among Movable Obstacles (PAMO), which seeks a minimum cost collision-free path among static obstacles from start to goal while allowing the robot to push away movable obstacles (i.e., objects) along its path when needed. To develop planners that are complete and optimal for PAMO, the planner has to search a giant state space involving both the location of the robot as well as the locations of the objects, which grows exponentially with respect to the number of objects. This paper leverages a simple yet under-explored idea that, only a small fraction of this giant state space needs to be searched during planning as guided by a heuristic, and most of the objects far away from the robot are intact, which thus leads to runtime efficient algorithms. Based on this idea, this paper introduces two PAMO formulations, i.e., bi-objective and resource constrained problems in an occupancy grid, and develops PAMO*, a planning method with completeness and solution optimality guarantees, to solve the two problems. We then further extend PAMO* to hybrid-state PAMO* to plan in continuous spaces with high-fidelity interaction between the robot and the objects. Our results show that, PAMO* can often find optimal solutions within a second in cluttered maps with up to 400 objects.

Keywords: Motion and Path Planning, Energy and Environment-Aware Automation, Deep Learning Methods
Abstract: Neural motion planners can increase motion plan- ning quality and, by reducing collision detection computations, improve runtime. However, when profiled on an accelerator-rich hardware system, neural planning contributes to more than 50% of the runtime, and 33% of the computation energy consumption, motivating the design of compute- and energy-efficient neural planners. In this work, we propose a neural planner using Binary Encoded Labels (BEL), where a set of binary classifiers are used instead of a typical regression network. Compared to conventional regression-based neural planners, the proposed BEL neural planner reduces neural planning (inference) computation and collision detection checks while maintaining equal or higher motion planning success rate across various motion planning benchmarks. This computation reduction can improve the energy efficiency of neural planning by 1.4x−21.4x. Finally, we demonstrate the trade-offs between collision detection and neural planning computation to maximize energy efficiency for different hardware configurations.

Keywords: Formal Methods in Robotics and Automation, Motion and Path Planning
Abstract: Navigating rigid body objects through crowded environments can be challenging, especially when narrow passages are presented. Existing sampling-based planners and optimization-based methods like mixed integer linear programming (MILP) formulations, suffer from limited scalability with respect to either the size of the workspace or the number of obstacles. In order to address the scalability issue, we propose a three-stage algorithm that first generates a graph of convex polytopes in the workspace free of collision, then poses a large set of small MILPs to generate viable paths between polytopes, and finally queries a pair of start and end configurations for a feasible path online. The graph of convex polytopes serves as a decomposition of the free workspace and the number of decision variables in each MILP is limited by restricting the subproblem within two or three free polytopes rather than the entire free region. Our simulation results demonstrate shorter online computation time compared to baseline methods and scales better with the size of the environment and tunnel width than sampling-based planners in both 2D and 3D environments.

Keywords: Distributed Robot Systems, Robust/Adaptive Control, Sensor-based Control
Abstract: Tethers are an underutilized tool in multi-robot systems: tethers can provide power, facilitate retrieval and sensing, and be used to manipulate and gather objects. Starting with the simplest possible configuration, our work explores how agents linked in series by flexible, passive, fixed-length tethers, can use those tethers as sensors to achieve distributed formation control. In this study, we extend upon previous work to show the applicability of strain-coordinated formation control for encapsulation and migration along a global gradient as well as the trade-offs between formation control and taxis in an obstacle-laden environment. Our results indicate significant potential for tethered robot collectives: versatile behaviors that can work on simple, resource-constrained robots or serve as a fallback mechanism in case more sophisticated means of coordination fail.

Keywords: Robotics in Hazardous Fields, Deep Learning for Visual Perception, Data Sets for Robotic Vision
Abstract: Long-term monitoring and exploration of extreme environments, such as underwater storage facilities, is costly, labor-intensive, and hazardous. Automating this process with low-cost, collaborative robots can greatly improve efficiency. These robots capture images from different positions, which must be processed simultaneously to create a spatio-temporal model of the facility. In this paper, we propose a novel approach that integrates data simulation, a multi-modal deep learning network for coordinate prediction, and image reassembly to address the challenges posed by environmental disturbances causing drift and rotation in the robots’ positions and orientations. Our approach enhances the precision of alignment in noisy environments by integrating visual information from snapshots, global positional context from masks, and noisy coordinates. We validate our method through extensive experiments using synthetic data that simulate real-world robotic operations in underwater settings. The results demonstrate very high coordinate prediction accuracy and plausible image assembly, indicating the real-world applicability of our approach. The assembled images provide clear and coherent views of the underwater environment for effective monitoring and inspection, showcasing the potential for broader use in extreme settings, further contributing to improved safety, efficiency, and cost reduction in hazardous field monitoring.

Keywords: Swarm Robotics, Embedded Systems for Robotic and Automation, Hardware-Software Integration in Robotics
Abstract: Swarm robotics focuses on designing and coordinating large groups of relatively simple robots to perform tasks in a decentralised and collective manner. The swarm provides a resilient and flexible solution for many applications. However, contemporary swarm robots have a significant power problem in that secondary (i.e. rechargeable) batteries are slow to charge and offer lifetimes of only a few years, increasing maintenance costs and pollution due to battery replacement.We imagine a different future, wherein battery free robots powered by supercapacitors can be recharged in seconds, offer long-life autonomous operation and can rapidly pass charge between one another using trophallaxis. In pursuit of this vision, we contribute the CapBot, a battery-free swarm robot equipped with Mecanum wheels, a Cortex M4F application processor and Bluetooth Low Energy networking. The CapBot fully recharges in 16 s, offers 51 min of autonomous operation at top speed, and can transfer up to 50% of its available charge to a peer via trophallaxis in under 20 s. The CapBot is fully open source and all software and hardware source is available online.

Keywords: Swarm Robotics, Social HRI, Art and Entertainment Robotics
Abstract: Robot swarms have the potential to help groups of people with social tasks, given their ability to scale to large numbers of robots and users. Developing multi-human-swarm interaction is therefore crucial to support multiple people interacting with the swarm simultaneously - which is an area that is scarcely researched, unlike single-human, single-robot or single-human, multi-robot interaction. Moreover, most robots are still confined to laboratory settings. In this paper, we present our work with MOSAIX, a swarm of robot Tiles, that facilitated ideation at a science museum. 63 robots were used as a swarm of smart sticky notes, collecting input from the public and aggregating it based on themes, providing an evolving visualization tool that engaged visitors and fostered their participation. Our contribution lies in creating a large-scale (63 robots and 294 attendees) public event, with a completely decentralized swarm system in real-life settings. We also discuss learnings we obtained that might help future researchers create multi-human-swarm interaction with the public.

Keywords: Software-Hardware Integration for Robot Systems, Aerial Systems: Applications, Swarm Robotics
Abstract: Efficient energy management and scalability are critical for aerial robots in tasks such as pickup-and-delivery and surveillance. This paper introduces MochiSwarm, an open-source testbed of light-weight micro robotic blimps designed for multi-robot operation without external localization. We propose a modular system architecture that integrates adaptable hardware, a flexible software framework, and a detachable perception module. The hardware is designed to allow for rapid modifications and sensor integration, while the software supports multiple actuation models and robust communication between a base station and multiple blimps. We showcase a differential-drive module as an example, in which autonomy is enabled by visual servoing using the perception module. A case study of pickup-and-delivery tasks with up to 12 blimps highlights the autonomy of the MochiSwarm without relying on external infrastructures.

Keywords: Aerial Systems: Perception and Autonomy, Micro/Nano Robots, Multi-Robot Systems
Abstract: Recent advances have improved autonomous navigation and mapping under payload constraints, but current multi-robot inspection algorithms are unsuitable for nano-drones, due to their need for heavy sensors and high computational resources. To address these challenges, we introduce ExploreBug, a novel hybrid frontier range-bug algorithm designed to handle limited sensing capabilities for a swarm of nano-drones. This system includes three primary components: a mapping subsystem, an exploration subsystem, and a navigation subsystem. Additionally, an intra-swarm collision avoidance system is integrated to prevent collisions between drones. We validate the efficacy of our approach through extensive simulations and real-world exploration experiments, involving up to seven drones in simulations and three in real-world settings, across various obstacle configurations and with a maximum navigation speed of 0.75 m/s. Our tests prove that the algorithm efficiently completes exploration tasks, even with minimal sensing, across different swarm sizes and obstacle densities. Furthermore, our frontier allocation heuristic ensures an equal distribution of explored areas and paths traveled by each drone in the swarm. We publicly release the source code of the proposed system to foster further developments in mapping and exploration using autonomous nano drones.

Keywords: Swarms, Path Planning for Multiple Mobile Robots or Agents, Distributed Robot Systems, Shape Formation
Abstract: Despite their growing popularity, swarms of robots remain limited by the operating time of each individual. We present algorithms which allow a human to sculpt a swarm of robots into a shape that persists in space perpetually, independent of onboard energy constraints such as batteries. Robots generate a path through a shape such that robots cycle in and out of the shape. Robots inside the shape react to human initiated changes and adapt the path through the shape accordingly. Robots outside the shape recharge and return to the shape so that the shape can persist indefinitely. The presented algorithms communicate shape changes throughout the swarm using message passing and robot motion. These algorithms enable the swarm to persist through any arbitrary changes to the shape. We describe these algorithms in detail and present their performance in simulation and on a swarm of mobile robots. The result is a swarm behavior more suitable for extended duration, dynamic shape-based tasks in applications such as entertainment, agriculture, and emergency response.

Keywords: Calibration and Identification, Redundant Robots, Kinematics
Abstract: Previous calibration techniques often relied on specialized end-effector tracking devices, such as a laser tracker, which can be expensive and impractical in specific environments. Furthermore, research on the calibration of redundant manipulators has been relatively scarce compared to non-redundant counterparts. To overcome these limitations, this article introduces a novel method for kinematic calibration of a damaged redundant serial robot, employing an indicated competitive swarm optimization with a finite-screw deviation model. The proposed kinematic calibration method utilizes a kinematic closed-loop method, which identifies an axis deviation without using expensive end-effector tracking equipment. Moreover, a competitive-swarm-inspired optimization model is introduced to efficiently identify axis deviations, significantly reducing the required calibration points compared to prior studies and thereby facilitating calibration for redundant manipulators. Both simulation and experiment were conducted to validate the proposed method using a seven-degree-of-freedom redundant serial robot. The results demonstrate the proposed calibration method's effectiveness and practicality, which can be easily applied for a redundant robot calibration.

Keywords: Calibration and Identification, Sensor Fusion, Deep Learning Methods
Abstract: In autonomous driving and robotic navigation, multi-sensor fusion technology has become increasingly mainstream, with precise sensor calibration as its foundation. Traditional calibration methods rely on manual effort or specific targets, limiting adaptability to complex environments. Learning-based calibration methods still face challenges, such as insufficient overlap between the fields of view (FoV) of multiple sensors and suboptimal cross-modal feature association, which hinder accurate parameter regression. Unlike traditional CNN-based networks, we propose a KansFormer-based self-Calibration Network for camera and LiDAR (KFCalibNet) that replaces fixed activation functions and linear transformations with learnable nonlinear activation functions. This enables the extraction of more fine-grained features from both image and point cloud, significantly enhancing the network's robustness in scenarios with limited FoV overlap. We also employ a multihead attention (MHA) module to compute correlations between image and point cloud features, significantly enhancing cross-modal feature association. To reduce learning complexity, we designed KansFormer with FastKAN as the feedforward network, enabling deep fusion and regression of fine-grained cross-modal features for accurate extrinsic calibration. KFCalibNet achieves an absolute average calibration error of 0.0965 cm in translation and 0.0234° in rotation on the KITTI Odometry dataset, outperforming existing state-of-the-art calibration methods. Moreover, its accuracy and generalization capability have been validated across multiple real-world railway lines.

Keywords: Dynamics, Deep Learning Methods, Calibration and Identification
Abstract: Among the many challenges of parallel kinematic manipulators, achieving high-speed and accurate control remains crucial. Estimating their dynamic properties is essential for designing precise and efficient control schemes. Conventional methods for dynamic model identification have been effective, though deep learning approaches have historically faced limitations due to data inefficiencies. However, recent advancements in physics-informed neural networks (PINNs) offer a way to improve both control and the extraction of interpretable physical properties from these robots. In this work, we propose and validate a PINN-based dynamic model for a Delta parallel robot, specifically the ABB IRB 360-6/1600. Our approach incorporates known physical properties, such as mass matrix sparsity, to improve accuracy and computational efficiency in dynamic model identification. To the best of our knowledge, this is the first study applying PINNs to model parallel robots. The method is validated experimentally, and its performance is compared to a validated identification technique for physically consistent identification, demonstrating the effectiveness of this approach for real-world applications in parallel robots.

Keywords: Calibration and Identification, Sensor Fusion, Localization
Abstract: This paper presents a study on the infield self-calibration of two rigidly connected IMUs' intrinsic parameters, without the aid of any external sensors, equipment, or specialized procedures. Specifically, we consider the calibration of gyroscope biases, gyroscope scale factors, and accelerometer biases, using only IMU data and known extrinsics between the two IMUs. We focus on the observability analysis of this system, and show that all gyroscope intrinsic parameters and a portion of accelerometer biases are observable, with information from both IMUs and sufficient motion. Moreover, we identify the additional unobservable directions in the intrinsic parameters that arise from various degenerate motions. Finally, we validate our observability findings through numerical simulations, and assess our system's calibration accuracy using real-world data.

Keywords: Calibration and Identification, RGB-D Perception, Perception for Grasping and Manipulation
Abstract: Hand-eye calibration is an important task in vision-guided robotic systems and is crucial for determining the transformation matrix between the camera coordinate system and the robot end-effector. Existing methods, for multi-view robotic systems, usually rely on accurate geometric models or manual assistance, generalize poorly, and can be very complicated and inefficient. Therefore, in this study, we propose PlaneHEC, a generalized hand-eye calibration method that does not require complex models and can be accomplished using only depth cameras, which achieves the optimal and fastest calibration results using arbitrary planar surfaces like walls and tables. PlaneHEC introduces hand-eye calibration equations based on planar constraints, which makes it strongly interpretable and generalizable. PlaneHEC also uses a comprehensive solution that starts with closed-form solution and improves it with iterative optimization, which greatly improves accuracy. We comprehensively evaluated the performance of PlaneHEC in both simulated and real-world environments and compared the results with other point-cloud-based calibration methods, proving its superiority. Our approach achieves universal and fast calibration with an innovative design of computational models, providing a strong contribution to the development of multi-agent system and embodied intelligence.

Keywords: Calibration and Identification, Kinematics
Abstract: This paper develops a Bayesian optimal experimental design for robot kinematic calibration on {mathbb{S}^3 !times! mathbb{R}^3}. Our method builds upon a Gaussian process approach that incorporates a geometry-aware kernel based on Riemannian Mat'ern kernels over {mathbb{S}^3}. To learn the forward kinematics errors via Bayesian optimization with a Gaussian process, we define a geodesic distance-based objective function. Pointwise values of this function are sampled via noisy measurements taken using fiducial markers on the end-effector using a camera and computed pose with the nominal kinematics. The corrected Denavit-Hartenberg parameters are obtained using an efficient quadratic program that operates on the collected data sets. The effectiveness of the proposed method is demonstrated via simulations and calibration experiments on NASA's ocean world lander autonomy testbed (OWLAT).

Keywords: Calibration and Identification, Deep Learning Methods, Sensor Fusion
Abstract: Sensor setups of robotic platforms commonly include both camera and LiDAR as they provide complementary information. However, fusing these two modalities typically requires a highly accurate calibration between them. In this paper, we propose MDPCalib which is a novel method for camera-LiDAR calibration that requires neither human supervision nor any specific target objects. Instead, we utilize sensor motion estimates from visual and LiDAR odometry as well as deep learning-based 2D-pixel-to-3D-point correspondences that are obtained without in-domain retraining. We represent camera-LiDAR calibration as an optimization problem and minimize the costs induced by constraints from sensor motion and point correspondences. In extensive experiments, we demonstrate that our approach yields highly accurate extrinsic calibration parameters and is robust to random initialization. Additionally, our approach generalizes to a wide range of sensor setups, which we demonstrate by employing it on various robotic platforms including a self-driving perception car, a quadruped robot, and a UAV. To make our calibration method publicly accessible, we release the code on our project website at https://calibration.cs.uni-freiburg.de.

Keywords: Legged Robots, Deep Learning Methods
Abstract: This paper introduces a novel proprioceptive state estimator for legged robots that combines model-based filters with deep neural networks. In environments where vision systems are not reliable, proprioceptive state estimators become indispensable. Traditionally, proprioceptive state estimators are based on model-based approaches, which rely solely on contact foot kinematics as measurements. In contrast, learning-based approaches have obtained new measurements, such as displacement and covariance, by leveraging real-world data in a supervised manner. In this work, we develop a state estimation framework that trains a neural measurement network (NMN) to estimate the base's linear velocity and foot contact probability, which are then employed as measurements in an invariant extended Kalman filter. Our approach relies solely on simulation data for training, as it allows us to obtain extensive data easily. We address the sim-to-real gap by adapting existing learning techniques and regularization. To validate our proposed method, we conduct hardware experiments using a quadruped robot on four types of terrain: flat, debris, soft, and slippery. In our experiments, the proposed method demonstrates significant improvements over the model-based state estimator, achieving an average reduction in Absolute Trajectory Error (ATE) by 61.8% for position and 8.5% for velocity.

Keywords: Legged Robots, Model Learning for Control, Calibration and Identification
Abstract: Accurate inertial parameter identification is crucial for the simulation and control of robots encountering intermittent contacts with the environment. Classically, robots' inertial parameters are obtained from CAD models that are not precise (and sometimes not available, e.g., Spot from Boston Dynamics), hence requiring identification. To do that, existing methods require access to contact force measurement, a modality not present in modern quadruped and humanoid robots. This paper presents an alternative technique that utilizes joint current/torque measurements —a standard sensing modality in modern robots— to identify inertial parameters without requiring direct contact force measurements. By projecting the whole-body dynamics into the null space of contact constraints, we eliminate the dependency on contact forces and reformulate the identification problem as a linear matrix inequality that can handle physical and geometrical constraints. We compare our proposed method against a common black-box identification method using a deep neural network and show that incorporating physical consistency significantly improves the sample efficiency and generalizability of the model. Finally, we validate our method on the Spot quadruped robot across various locomotion tasks, showcasing its accuracy and generalizability in real-world scenarios over different gaits.

Keywords: Legged Robots, Dynamics, Multi-Contact Whole-Body Motion Planning and Control
Abstract: Accurate estimation of contact forces is crucial for the effective control of quadrupedal robots, especially in complex locomotion scenarios. In this paper, we introduce a novel force estimation technique for robots equipped with transformable leg-wheels. Unlike conventional methods that focus on forces at specific contact points, our approach expresses varying contact points through a simplified kinematic model and derives the corresponding Jacobian matrices. This allows us to apply the virtual work method to evaluate contact forces across the entire surface of the leg-wheel, including the tips, sides, and other contact regions. This adaptability is particularly advantageous in hybrid locomotion modes, where different parts of the leg-wheel interact with the terrain. The proposed method is highly efficient, relying solely on motor current and position feedback without the need for additional sensors. We validate our approach through simulations and real-world experiments, demonstrating its accuracy, robustness, and applicability under diverse operational conditions.

Keywords: Legged Robots, Motion Control
Abstract: In this paper we address the simultaneous collision detection and force estimation problem for quadrupedal loco- motion using joint encoder information and the robot dynamics only. We design an interacting multiple-model Kalman filter (IMM-KF) that estimates the external force exerted on the robot and multiple possible contact modes. The method is invariant to any gait pattern design. Our approach leverages pseudo-measurement information of the external forces based on the robot dynamics and encoder information. Based on the estimated contact mode and external force, we design a reflex motion and an admittance controller for the swing leg to avoid collisions by adjusting the leg’s reference motion. Additionally, we implement a force-adaptive model predictive controller to enhance balancing. Simulation ablatation studies and experiments show the efficacy of the approach.

Keywords: Legged Robots, Sensorimotor Learning, Mapping
Abstract: In critical applications, including search-and- rescue in degraded environments, blockages can be prevalent and prevent the effective deployment of certain sensing modalities, particularly vision, due to occlusion and the constrained range of view of onboard camera sensors. To enable robots to tackle these challenges, we propose a new approach, Proprioceptive Obstacle Detection and Estimation while navigating in clutter (PROBE), which instead relies only on the robot’s proprioception to infer the presence or absence of occluded rectangular obstacles while predicting their dimensions and poses in SE(2). The proposed approach is a Transformer neural network that receives as input a history of applied torques and sensed whole-body movements of the robot and returns a parameterized representation of the obstacles in the environment. The effectiveness of PROBE is evaluated on simulated environments in Isaac Gym and with a real Unitree Go1 quadruped robot. The project webpage can be found at https://dhruvmetha.github.io/legged-probe/

Keywords: Legged Robots, Sensor Fusion
Abstract: In this paper, we present a fast and decentralized state estimation framework for the control of legged locomotion. The nonlinear estimation of the floating base states is decentralized to an orientation estimation via Extended Kalman Filter (EKF) and a linear velocity estimation via Moving Horizon Estimation (MHE). The EKF fuses the inertia sensor with vision to estimate the floating base orientation. The MHE uses the estimated orientation with all the sensors within a time window in the past to estimate the linear velocities based on a time-varying linear dynamics formulation of the interested states with state constraints. More importantly, a marginalization method based on the optimization structure of the full information filter (FIF) is proposed to convert the equality-constrained FIF to an equivalent MHE. This decoupling of state estimation promotes the desired balance of computation efficiency, accuracy of estimation, and the inclusion of state constraints. The proposed method is shown to be capable of providing accurate state estimation to several legged robots, including the highly dynamic hopping robot PogoX, the bipedal robot Cassie, and the quadrupedal robot Unitree Go1, with a frequency at 200 Hz and a window interval of 0.1s.

Keywords: Physically Assistive Devices, Domestic Robotics, Mechanism Design
Abstract: As over 11,000 people turn 65 each day in the U.S., our country, like many others, is facing growing challenges in caring for elderly persons, further exacerbated by a major shortfall of care workers. To address this, we introduce an eldercare robot (E-BAR) capable of lifting a human body, assisting with postural changes/ambulation, and catching a user during a fall, all without the use of any wearable device or harness. Our robot is the first to integrate these 3 tasks, and is capable of lifting the full weight of a human outside of the robot’s base of support (across gaps and obstacles). In developing E-BAR, we interviewed nurses and care professionals and conducted user-experience tests with elderly persons. Based on their functional requirements, the design parameters were optimized using a computational model and trade-off analysis. We developed a novel 18-bar linkage to lift a person from a floor to a standing position along a natural trajectory, while providing maximal mechanical advantage at key points. An omnidirectional, non-holonomic drive base, in which the wheels could be oriented to passively maximize floor grip, enabled the robot to resist lateral forces without active compensation. With a minimum width of 38 cm, the robot’s small footprint allowed it to navigate the typical home environment. Four airbags were used to catch and stabilize a user during a fall in less than 250 ms. We demonstrate E-BAR's utility in multiple typical home scenarios, including getting into/out of a bathtub, bending to reach for objects, sit-to-stand transitions, and ambulation.

Keywords: Wearable Robotics, AI-Based Methods, Health Care Management
Abstract: People with visual impairments rely on Electronic Travel Aids (ETAs), such as sensor-equipped guide canes, for safe and effective navigation. Misalignment or improper handling of these devices can reduce their effectiveness, increasing the risk of collisions and injuries. This paper presents an AI-based embedded system designed to predict and correct the orientation of a guide cane in real time. By integrating an Inertial Measurement Unit (IMU) with a neural network, the system continuously monitors the cane's lateral angle and orientation while providing feedback to help the user self-correct. The feedback is proportional to the degree of error, guiding users to maintain proper cane positioning during mobility. The device logs data that can be visualized remotely, offering mobility trainers valuable insights into the user's navigation patterns. Evaluation by visually impaired users demonstrated that the system effectively aided in real-time orientation correction. This system represents a significant advancement in improving the safety and independence of individuals with visual impairments through wearable ETAs.

Keywords: Wearable Robotics, Physically Assistive Devices, Human and Humanoid Motion Analysis and Synthesis
Abstract: Supernumerary Robotic Limbs (SRLs) can assist human motions by providing extra degrees of freedom (DoFs) and body support. The extra DoFs lead to larger design space in structure and control policies, which is complex and timeconsuming with the traditional manual design process. In this pilot study, we proposed a novel morphology-controller cooptimization framework to automatically generate and optimize the SRL structure based on the locomotion task input. There are two layers, with the inner layer optimizing the controller to achieve human-robot synchronization, and the outer layer optimizing the morphology parameters for performance enhancement. We validated the proposed framework through simulations using SRLs in a load-bearing locomotion task. The results demonstrate that the controller optimization can automatically generate realistic gait patterns and stable humanrobot synchronization, while the SRLs significantly improve the user’s load-bearing capability. Additionally, the co-optimization process reduces both the manufacturing cost of the SRL and the torque on the joints. This approach shows potential for exhaustive exploration of the design space and acceleration of the design process. Future works will be done in a more realistic SRL generative design model and achieve Sim2Real for practical uses.

Keywords: Physically Assistive Devices, Rehabilitation Robotics, Machine Learning for Robot Control
Abstract: In this paper, we present an adaptive walker system designed to address limitations in current intelligent walker technologies. While recent advancements have been made in this field, existing systems often struggle to seamlessly interpret user intent for speed control and lack adaptability across diverse scenarios and terrain. Our proposed solution incorporates high-resolution tactile sensors, deep learning algorithms, IMU sensors, and linear motors to dynamically adjust to the user’s intentions and terrain changes. The system is capable of predicting the user’s desired speed with an error margin of only 20.99%, relying solely on tactile input from hand and arm contact points. Additionally, it maintains the walker’s horizontal stability with an error of less than 1 degree by adjusting leg lengths in response to variations in ground angle. This adaptive walker enhances user safety and comfort, particularly for individuals with reduced strength or cognitive abilities, and offers reliable assistance on uneven terrain such as uphill and downhill paths.

Keywords: Human-Centered Robotics, Representation Learning, Imitation Learning
Abstract: Robotic assistive feeding holds significant promise for improving the quality of life for individuals with eating disabilities. However, acquiring diverse food items under varying conditions and generalizing to unseen food presents unique challenges. Existing methods that rely on surface-level geometric information (e.g., bounding box and pose) derived from visual cues (e.g., color, shape, and texture) often lacks adaptability and robustness, especially when foods share similar physical properties but differ in visual appearance. We employ imitation learning (IL) to learn a policy for food acquisition. Existing methods employ IL or Reinforcement Learning (RL) to learn a policy based on off-the-shelf image encoders such as ResNet-50. However, such representations are not robust and struggle to generalize across diverse acquisition scenarios. To address these limitations, we propose a novel approach, IMRL (Integrated Multi-Dimensional Representation Learning), which integrates visual, physical, temporal, and geometric representations to enhance the robustness and generalizability of IL for food acquisition. Our approach captures food types and physical properties (e.g., solid, semi-solid, granular, liquid, and mixture), models temporal dynamics of acquisition actions, and introduces geometric information to determine optimal scooping points and assess bowl fullness. IMRL enables IL to adaptively adjust scooping strategies based on context, improving the robot’s capability to handle diverse food acquisition scenarios. Experiments on a real robot demonstrate our approach’s robustness and adaptability across various foods and bowl configurations, including zero-shot generalization to unseen settings. Our approach achieves an improvement up to 35% in success rate compared with the best-performing baseline. More details can be found on our website https://ruiiu.github.io/imrl.

Keywords: Physically Assistive Devices, Physical Human-Robot Interaction, Human-Centered Robotics
Abstract: Our team developed a riding ballbot (called PURE) that is dynamically stable, omnidirectional, and driven by lean-to-steer control. A hands-free admittance control scheme (HACS) was previously integrated to allow riders with different torso functions to control the robot's movements via torso leaning and twisting. Such an interface requires motor coordination skills and could result in collisions with obstacles due to low proficiency. Hence, a shared controller (SC) that limits the speed of PURE could be helpful to ensure the safety of riders. However, the self-balancing dynamics of PURE could result in a weak control authority of its motion, in which the torso motion of the rider could easily result in poor tracking of the command speed dictated by the shared controller. Thus, we proposed an interactive hands-free admittance control scheme (iHACS), which added two modules to the HACS to improve the speed-tracking performance of PURE: control gain personalization module and interaction compensation module. Human riding tests of simple tasks, idle-keeping and speed-limiting, were conducted to compare the performance of HACS and iHACS. Two manual wheelchair users and two able-bodied individuals participated in this study. They were instructed to use ``adversarial" torso motions that would tax the SC's ability to keep the ballbot idling or below a set speed, i.e., competing objectives between rider and robot. In the idle-keeping tasks, iHACS demonstrated minimal translational motion and low command speed tracking RMSE, even with significant torso lean angles. During the speed-limiting task, where the commanded speed was saturated at 0.5 m/s, the system achieved an average maximum speed of 1.1 m/s with iHACS, compared with that of over 1.9 m/s with HACS. These results suggest that iHACS can enhance PURE's control authority over the rider, which enables PURE to provide physical interactions back to the rider and results in a collaborative rider-robot synergy.

Keywords: Physical Human-Robot Interaction, Model Learning for Control, Human-Centered Robotics
Abstract: Robots have the potential to assist people with disabilities and the elderly. One of the most common and burdensome tasks for caregivers is dressing. Two challenges of robot-assisted dressing are modeling the dynamics of garments and handling visual occlusions that obstruct the perception of the full state of the garment due to the proximity between the garment, the robot, and the human. In this paper, we propose a diffusion-based dynamics model for garments during robot-assisted dressing that can deal with partial point cloud observations. The diffusion model, conditioned on the observation and the robot's action, is used to predict a full point cloud of the garment's opening of the future state. The model is utilized in a model predictive controller, that is trained iteratively with model-based reinforcement learning. In our experiments, we examine a common problem of dressing: the insertion of a garment's sleeve on an arm. As demonstrated by the performed experiments, the proposed diffusion-based model predictive controller can be effectively used for robot-assisted dressing and handle visual occlusions. Moreover, our approach is highly sample-efficient. Specifically, the controller achieved 91.2% success rate in the examined dressing task with less than 100 sampled trajectories. Real-wold experiments demonstrate that the proposed method can adapt to the sim-to-real gap and generalize well to novel garments and configurations of the body.

Keywords: Data Sets for Robotic Vision, Vision-Based Navigation, Intelligent Transportation Systems
Abstract: With the rapid advancements in autonomous driving, accurately predicting pedestrian behavior has become essential for ensuring safety in complex and unpredictable traffic conditions. The growing interest in this challenge highlights the need for comprehensive datasets that capture unstructured environments, enabling the development of more robust prediction models to enhance pedestrian safety and vehicle navigation. In this paper, we introduce an Indian driving pedestrian dataset designed to address the complexities of modeling pedestrian behavior in unstructured environments, such as illumination changes, occlusion of pedestrians, unsignalized scene types and vehicle-pedestrian interactions. The dataset provides high-level and detailed low-level comprehensive annotations focused on pedestrians requiring the ego-vehicle’s attention. Evaluation of the state-of-the-art intention prediction methods on our dataset shows a significant performance drop of up to 15%, while trajectory prediction methods underperform with an increase of up to 1208 MSE, defeating standard pedestrian datasets. Additionally, we present exhaustive quantitative and qualitative analysis of intention and trajectory baselines. We believe that our dataset will open new challenges for the pedestrian behavior research community to build robust models. Project Page: https://cvit.iiit.ac.in/ research/projects/cvit-projects/iddped

Keywords: Intelligent Transportation Systems
Abstract: Accurate prediction of pedestrian trajectories is crucial as autonomous vehicles become more prevalent on roads. The dynamic nature of urban environments and the less predictable behavior of pedestrians present significant challenges in developing reliable prediction models. Earlier methods relying on recurrent neural networks (RNNs) and long-short-term memory (LSTM) networks have shown promise, but often fail to fully take advantage of the rich visual and contextual information available in real-world scenarios. Recent advances in vision-language models (VLMs) offer new opportunities to improve pedestrian trajectory prediction by incorporating multimodal reasoning capabilities. This paper introduces a novel approach that uses a powerful pre-trained VLM to improve the estimation of pedestrian trajectories. Specifically, we first enable learning of semantically useful scene context and high-level reasoning feature via vision-language model fine-tuning on specific prompts using road scenes with pedestrians. Next, with the learned VLM features and the pedestrian's past trajectory history, we predict future trajectories using an encoder-decoder head. Through experiments with first-person datasets JAAD and PIE, we show that utilizing visual-linguistic semantics via a pre-trained vision-language model outperforms previous methods in both deterministic and stochastic trajectory prediction setups.

Keywords: Intelligent Transportation Systems, Autonomous Vehicle Navigation, Autonomous Agents
Abstract: Trajectory prediction models in autonomous driving are vulnerable to perturbations from non-causal agents whose actions should not affect the ego-agent’s behavior. Such perturbations can lead to incorrect predictions of other agents’ trajectories, potentially compromising the safety and efficiency of the ego-vehicle’s decision-making process. Motivated by this challenge, we propose Causal tRajecTory predICtion (CRiTIC), a novel model that utilizes a causal discovery network to identify inter-agent causal relations over a window of past time steps. To incorporate discovered causal relationships, we propose a novel Causal Attention Gating mechanism to selectively filter information in the proposed Transformer-based architecture. We conduct extensive experiments on two autonomous driving benchmark datasets to evaluate the robustness of our model against non-causal perturbations and its generalization capacity. Our results indicate that the robustness of predictions can be improved by up to 54% without a significant detriment to prediction accuracy. Lastly, we demonstrate the superior domain generalizability of the proposed model, which achieves up to 29% improvement in cross-domain performance. These results underscore the potential of our model to enhance both robustness and generalization capacity for trajectory prediction in diverse autonomous driving domains.

Keywords: Visual Tracking, Visual Learning, Deep Learning for Visual Perception
Abstract: 3D multi-object tracking plays a critical role in autonomous driving by enabling the real-time monitoring and prediction of multiple objects' movements. Traditional 3D tracking systems are typically constrained by predefined object categories, limiting their adaptability to novel, unseen objects in dynamic environments. To address this limitation, we introduce open-vocabulary 3D tracking, which extends the scope of 3D tracking to include objects beyond predefined categories. We formulate the problem of open-vocabulary 3D tracking and introduce dataset splits designed to represent various open-vocabulary scenarios. We propose a novel approach that integrates open-vocabulary capabilities into a 3D tracking framework, allowing for generalization to unseen object classes. Our method effectively reduces the performance gap between tracking known and novel objects through strategic adaptation. Experimental results demonstrate the robustness and adaptability of our method in diverse outdoor driving scenarios. To the best of our knowledge, this work is the first to address open-vocabulary 3D tracking, presenting a significant advancement for autonomous systems in real-world settings.

Keywords: Object Detection, Segmentation and Categorization, Visual Tracking, Data Sets for Robotic Vision
Abstract: Events cameras are ideal sensors for enabling robots to detect and track objects in highly dynamic environments due to their low latency output, high temporal resolution, and high dynamic range. In this paper, we present the Asynchronous Event Multi-Object Tracking (AEMOT) algorithm for detecting and tracking multiple objects by processing individual raw events asynchronously. AEMOT detects salient event blob features by identifying regions of consistent optical flow using a novel Field of Active Flow Directions built from the Surface of Active Events. Detected features are tracked as candidate objects using the recently proposed Asynchronous Event Blob (AEB) tracker in order to construct small intensity patches of each candidate object. A novel learnt validation stage promotes or discards candidate objects based on classification of their intensity patches, with promoted objects having their position, velocity, size, and orientation estimated at their event rate. We evaluate AEMOT on a new Bee Swarm Dataset, where it tracks dozens of small bees with precision and recall performance exceeding that of alternative event-based detection and tracking algorithms by over 37%. Source code and the labelled event Bee Swarm Dataset will be open sourced.

Keywords: Computer Vision for Transportation, Sensor Fusion, Deep Learning Methods
Abstract: Vehicle-to-everything technologies (V2X) have become an ideal paradigm to extend the perception range and see through the occlusion. Exiting efforts focus on single-frame cooperative perception, however, how to capture the temporal cue between frames with V2X to facilitate the prediction task even the planning task is still underexplored. In this paper, we introduce the Co-MTP, a general cooperative trajectory prediction framework with multi-temporal fusion for autonomous driving, which leverages the V2X system to fully capture the interaction among agents in both history and future domains to benefit the planning. In the history domain, V2X can complement the incomplete history trajectory in single-vehicle perception, and we design a heterogeneous graph transformer to learn the fusion of the history feature from multiple agents and capture the history interaction. Moreover, the goal of prediction is to support future planning. Thus, in the future domain, V2X can provide the prediction results of surrounding objects, and we further extend the graph transformer to capture the future interaction among the ego planning and the other vehicles' intentions and obtain the final future scenario state under a certain planning action. We evaluate the Co-MTP framework on the real-world dataset V2X-Seq, and the results show that Co-MTP achieves state-of-the-art performance and that both history and future fusion can greatly benefit prediction. Our code is available on our project website: https://xiaomiaozhang.github.io/Co-MTP/

Keywords: Wheeled Robots, Collision Avoidance, Embedded Systems for Robotic and Automation
Abstract: Head-to-head racing against opponents is a challenging and emerging topic in the domain of autonomous racing. We propose Predictive Spliner, a data-driven overtaking planner designed to enhance competitive performance by anticipating opponent behavior. Using GP regression, the method learns and predicts the opponent’s trajectory, enabling the ego vehicle to calculate safe and effective overtaking maneuvers. Experimentally validated on a 1:10 scale autonomous racing platform, Predictive Spliner outperforms commonly employed overtaking algorithms by overtaking opponents at up to 83.1% of its own speed, being on average 8.4% faster than the previous best-performing method. Additionally, it achieves an average success rate of 84.5%, which is 47.6% higher than the previous best-performing method. The proposed algorithm maintains computational efficiency, making it suitable for real-time robotic applications. These results highlight the potential of Predictive Spliner to enhance the performance and safety of autonomous racing vehicles. The code for Predictive Spliner is available at: https://github.com/ForzaETH/predictive-spliner.

Keywords: Surgical Robotics: Steerable Catheters/Needles, Soft Robot Materials and Design, Micro/Nano Robots
Abstract: Recently, in the minimally invasive surgery field, magnetic soft continuum robots (MSCRs) have been actively studied, which are driven by an external magnetic field with a small magnet attached to the tip of the SCR. In addition, MSCR with opposite magnets (MSCR-OMs) has been reported for high dexterity, which has a small permanent magnet attached to the end of the MSCR and an additional opposite magnet fixed in the middle. To overcome the limitations of the existing MSCR and MSCR-OM and improve the workspace, we proposed a magnetic soft continuum robot with a movable opposite magnet (MSCR-MOM) with a 2.2 mm diameter and 10cm length, that can change the position of the opposite magnet. In this study, an analytical model of the proposed MSCR-MOM was presented, and through simulation and various experiments, its characteristics were analyzed and the workspace expansion was validated. In addition, the clinical applicability of the proposed MSCR-MOM was verified through phantom experiments. In the future, we expect that the proposed MSCR-MOM will be developed into a medical catheter that can be applied in various procedures through miniaturization and various clinical application studies.

Keywords: Medical Robots and Systems, Surgical Robotics: Steerable Catheters/Needles, Modeling, Control, and Learning for Soft Robots
Abstract: Robotic-assisted percutaneous coronary intervention (PCI) holds considerable promise for elevating precision and safety in cardiovascular procedures. Nevertheless, current systems heavily depend on human operators, resulting in variability and the potential for human error. To tackle these challenges, Sim4EndoR, an innovative reinforcement learning (RL) based simulation environment, is first introduced to bolster task-level autonomy in PCI. This platform offers a comprehensive and risk-free environment for the development, evaluation, and refinement of potential autonomous systems, enhancing data collection efficiency and minimizing the need for costly hardware trials. A notable aspect of the groundbreaking Sim4EndoR is its reward function, which takes into account the anatomical constraints of the vascular environment, utilizing the geometric characteristics of vessels to steer the learning process. By seamlessly integrating advanced physical simulations with neural network-driven policy learning, Sim4EndoR fosters efficient sim-to-real translation, paving the way for safer, more consistent robotic interventions in clinical practice, ultimately improving patient outcomes.

Keywords: Medical Robots and Systems, Surgical Robotics: Steerable Catheters/Needles, Tendon/Wire Mechanism
Abstract: Craniosynostosis involves premature fusion of the cranial sutures resulting in abnormal skull morphology and elevated intracranial pressure. Surgical intervention is necessary to correct the skull shape and to allow for unrestricted brain growth. This study presents a novel snake robot designed for minimally invasive cranial osteotomies featuring two articulating bending segments. The end-effector comprises a bone-punch for bone-cutting, a dural and scalp retractor, as well as channels for an endoscope and an instrument. The robot’s bending mechanism is driven by tendons and utilizes geared linkages to facilitate a smooth curved shape. Pre-tensioned antagonistic tendons allow the robot to modulate its stiffness to adapt to external loads. A follow-the-leader algorithm was implemented to guide the robot along a skull cutting path. Experimental results demonstrated that at maximum bending of 60◦ for segment 1 and 90◦ for segment 2 there was a 15.9◦ and 11.5◦ error, respectively. Position errors ranged from 2.5 to 21.5mm when tracing a curved path. The tool increased stiffness with tendon pre-tensioning from 20-100N during bent configurations q1 and q2 for segments 1 and 2, respectively, at [q1,q2] = [0◦,30◦] and [30◦,60◦]. Tip deflection reduced from 0.42 to 0.03cm and 0.37 to 0.10cm during axial loading and from 11.40 to 3.88cm and 3.62 to 0.48cm during radial loading for each configuration, respectively. Ex vitro trials demonstrated the robots ability to perform simulated osteotomies on skull models to 68-73% of desired path lengths with a maximum deviation of 8mm.

Keywords: Medical Robots and Systems, Surgical Robotics: Steerable Catheters/Needles, Soft Sensors and Actuators
Abstract: To address the challenges associated with shape sensing of continuum manipulators (CMs) using Fiber Bragg Grating (FBG) optical fibers, we present a unique shape sensing assembly utilizing solely a single Optical Frequency Domain Reflectometry (OFDR) fiber attached to a flat nitinol wire (NiTi).Integrating this easy-to-manufacture unique sensor with a long and soft CM with 170 mm length, we performed different experiments to evaluate its C-, J-, and S-shape reconstruction ability. Results demonstrate phenomenal shape reconstruction accuracy for the performed C-shape (< 3.14 mm tip error, < 2.54 mm shape error), J-shape (< 1.91 mm tip error, < 1.11 mm shape error), and S-shape (< 1.74 mm tip error, < 1.40 mm shape error) experiments.

Keywords: Medical Robots and Systems, Surgical Robotics: Steerable Catheters/Needles, Haptics and Haptic Interfaces
Abstract: Knowledge of the tip contact force in continuum robots, which are often used as medical instruments, is critical for clinical applications. It enhances the interventionalist's decision-making, navigation efficiency, and procedural safety. However, accurately determining the tip contact force in conventionally sized instruments remains challenging. This study introduces a learning-based method for estimating the external contact force at the tip of a continuum robot. By leveraging curvature and bending angle data from a multi-core fiber equipped with fiber Bragg gratings (FBGs) embedded inside the Nitinol tube, the method maps these inputs to the corresponding tip force in 3D. Experiments conducted on an FBG-embedded Nitinol rod validate the feasibility of the proposed method, yielding Mean Squared Error (MSE), Mean Absolute Error (MAE), and Root Mean Squared Error (RMSE) values of 20.9 (mN²), 2.7 (mN), and 4.6 (mN), respectively, which represent a 26% improvement compared to the learning-based vision methodology.

Keywords: Medical Robots and Systems, Force and Tactile Sensing
Abstract: In endoluminal surgeries, inserting a flexible endoscope is one of the fundamental procedures. During this process, vision remains the primary feedback, while the perception of tactile magnitude and location is insufficient. This limitation can hinder the clinician’s efficiency when navigating the endoscope through various segments of the natural lumens. To address this issue, we propose a fiber Bragg grating (FBG)–based tissue-compliant sensor cap with multi-mode sensing capabilities, including contact location identification at the terminal surface and the three-dimensional contact force perception at the tip. The soft sensor cap can be affixed to the standard endoscope tip, like a distal attachment cap, for easy installation. Utilizing the relative contact location information, operators can adjust the steerable segment of the endoscope when transitioning from one segment of a natural orifice to a narrower segment, which may be obstructed by constricted lumens. A finite element analysis simulation and the corresponding calibration process based on learning-based approaches have been carried out. The FBG-based sensor can perceive the tip contact force and identify the axial contact location with high precision, where the force perception error is less than 3%, and the contact location identification accuracy is 98.8%. The experimental results demonstrate the potential of the proposed sensing mechanism to be applied in surgeries requiring endoscope insertions.

Keywords: Surgical Robotics: Steerable Catheters/Needles, Modeling, Control, and Learning for Soft Robots, Optimization and Optimal Control
Abstract: This study focuses on the flexible tube of a bronchoscope robot used in pulmonary intervention surgery, which is considered as a continuum robot. The dynamics model is proposed based on the Koopman operator, leveraging real data to solve for the system matrix parameters accurately. To enhance control precision, we designed a model predictive control (MPC) algorithm aimed at tracking the desired curvature and deflection angles of the flexible tube. The MPC controller uses real-time data from electromagnetic sensors to adjust the tube shape, ensuring accurate and responsive manipulation. The effectiveness of the proposed algorithm are validated through extensive experiments conducted on the Binary experimental platform, demonstrating significant improvements in tracking performance and operational reliability compared to traditional open-loop control methods.

Keywords: Compliance and Impedance Control, Human-Robot Collaboration, Motion Control
Abstract: Applications involving serial manipulators, in both co-manipulation with humans and autonomous operation tasks, require the robot to render high admittance so as to minimize contact forces and maintain stable contacts with high-stiffness surfaces. This can be achieved through admittance control, however, inner loop dynamics limit the bandwidth within which the desired admittance can be rendered from the outer loop. Moreover, perturbations affect the admittance control performance whereas other system specific limitations such as “black box” PD position control in typical industrial manipulators hinder the implementation of more advanced control methods. To address these challenges, a perturbation-robust multisensor framework, designed for serial manipulators engaged in contact-rich tasks involving heavy payloads, is introduced in this paper. Within this framework, a generalized perturbation-robust observer (PROB), which exploits the joint velocity measurements and inner loop velocity control model, and accommodates the varying stiffness of contacts through contact force measurements is introduced. Three PROBs including a novel Combined Dynamics Observer (CDYOB) are presented. The CDYOB can render wide-range admittance without bandwidth limitations from the inner loop. Theoretical analyses and experiments with an industrial robot validate the effectiveness of the proposed method.

Keywords: Tendon/Wire Mechanism, Redundant Robots, Mechanism Design
Abstract: Hyper-redundant manipulators have been developed for hazardous environment exploration due to their flexibility and high agility in workplace. In this research, we designed a hyper-redundant manipulator by integrating Twisted String Actuators (TSAs) and Rolling Contact Joints (RCJs) to overcome the limitations of traditional cable-driven system, such as difficulties with long-distance power transmission, and to achieve high payload capability with a compact design. To prevent instantaneous tension loss by the slack and to enhance control consistency of the manipulator by preserving the relationship between contraction ratio of TSA and motor rotations, we proposed a tension maintenance mechanism using compression springs at the distal end of the manipulator. Additionally, to reduce losses from string contact friction, spring sheaths were inserted along the joint holes. Our approaches enhance the repeatability and position controllability of the manipulator. We noted a 33.5% reduction of error in repeatability test along with 35.9% and 38.8% improvements in piecewise position control accuracy and precision compared to a conventional manipulator, respectively, leading to enhanced controllability. We also experimentally verified that the proposed manipulator can maintain its trajectory with a variance of less than 2.83% up to 1600 g. Overall, our manipulator has the potential to expand the exploration environments in which robots can be used by simultaneously demonstrating large payload and controllability.

Keywords: Compliance and Impedance Control, Force Control, Performance Evaluation and Benchmarking
Abstract: Over the last decade, industrial robots have evolved from well-established position-controlled systems to collaborative and soft robots. In 2017 we introduced the tactile lightweight robot system Franka Emika Robot, characterized by advanced safety control, force sensing, joint torque and force control, and hand-guiding performance. In the meantime, the system has become a well-adopted reference platform for robotics research in AI and machine learning, manipulation, control, human-robot interaction, and motion planning. It features multiple functional and widely used interfaces, including 1kHz real-time joint torque control access or precise kinematic and dynamic models. Furthermore, it became a crystallization point of a research ecosystem since the system's affordability further lowered the entrance barrier to high-performance joint torque-controlled robots. In this article, a quantitative analysis and discussion of the use of the system in worldwide research labs over the last five years, its impact on the creation of a compatible software ecosystem, and examples of milestone experiments made possible with the robot are given. The robotics community benefits from understanding

Keywords: Learning from Demonstration, Constrained Motion Planning
Abstract: An open problem in industrial automation is to reliably perform tasks requiring in-contact movements with complex workpieces, as current solutions lack the ability to seamlessly adapt to the workpiece geometry. In this paper, we propose a Learning from Demonstration approach that allows a robot manipulator to learn and generalise motions across complex surfaces by leveraging differential mathematical operators on discrete manifolds to embed information on the geometry of the workpiece extracted from triangular meshes, and extend the Dynamic Movement Primitives (DMPs) framework to generate motions on the mesh surfaces. We also propose an effective strategy to adapt the motion to different surfaces, by introducing an isometric transformation of the learned forcing term. The resulting approach, namely MeshDMP, is evaluated both in simulation and real experiments, showing promising results in typical industrial automation tasks like car surface polishing.

Keywords: Engineering for Robotic Systems, Mobile Manipulation, Perception for Grasping and Manipulation
Abstract: This paper presents a fully autonomous robotic system that performs sim-to-real transfer in complex long-horizon tasks involving navigation, recognition, grasping, and stacking in an environment with multiple obstacles. The key feature of the system is the ability to overcome typical sensing and actuation discrepancies during sim-to-real transfer and to achieve consistent performance without any algorithmic modifications. To accomplish this, a lightweight noise-resistant visual perception system and a nonlinearity-robust servo system are adopted. We conduct a series of tests in both simulated and real-world environments. The visual perception system achieves the speed of 11 ms per frame due to its lightweight nature, and the servo system achieves sub-centimeter accuracy with the proposed controller. Both exhibit high consistency during sim-to-real transfer. Our robotic system took first place in the mineral searching task of the Robotic Sim2Real Challenge hosted at ICRA 2024. The simulator is available from the competition committee at https://github.com/AIR-DISCOVER/ICRA2024-Sim2Real-RM, and all code and competition videos can be accessed via our GitHub repository at https://github.com/Bob-Eric/rmus2024_solution_ZeroBug.

Keywords: Deep Learning for Visual Perception, Grasping, Data Sets for Robotic Vision
Abstract: The interaction of robots with their environment requires robust object-centric perception capabilities, typically achieved using learning-based methods trained on synthetic data. However, real-world deployment demands evaluating these capabilities in relevant environments, often involving extensive manual annotation for a quantitative analysis. Additionally, standardized evaluations for robotic tasks, such as grasping, need reproducible object scene configurations and performance benchmarks. We propose a solution to both problems by temporarily employing 3D-printed components, so-called fixtures, which can be designed for any rigid object. Once the scene is set up and object poses are extracted, the fixtures are removed, leaving the natural scene without any artificial distractions. The presented approach is seemingly applicable for pre-determined configurations of multiple objects, which enables precise re-building of scenes with consistent object-to-object relations. Our suggested annotation procedure achieves strong pose accuracy solely on RGB images without any manual involvement. We evaluate and show the usability of the proposed fixtures for automated real-world data annotation to fine-tune a detector and for benchmarking object pose estimation algorithms for robotic grasping. Code and fixture meshes for 3D printing are available at https://github.com/DLR-RM/fixture_generation.

Keywords: Optimization and Optimal Control, Sensor-based Control, Motion Control
Abstract: Model predictive control (MPC) has become increasingly popular for the control of robot manipulators due to its improved performance compared to instantaneous control approaches. However, tuning these controllers remains a significant hurdle. To address this hurdle, we propose a practical MPC formulation which retains the more interpretable tuning parameters of the instantaneous control approach while enhancing performance through a prediction horizon. The formulation is motivated at hand of a simple example, highlighting the practical tuning challenges associated with typical MPC approaches and showing how the proposed formulation alleviates these challenges. Furthermore, the formulation is validated on a surface-following task, illustrating its applicability to industrially relevant scenarios. Although the research is presented in the context of robot manipulator control, we anticipate that the formulation is more broadly applicable.

Keywords: Compliance and Impedance Control, Compliant Joints and Mechanisms, Prosthetics and Exoskeletons
Abstract: In addition to lifting up the body during gait, human legs provide stabilizing torques that can be modeled as a spring-damper mechanical impedance. While powered prosthetic leg actuators can also imitate spring-damper be- haviors, the rendered impedance can be quite different from the desired impedance, stemming from unmodeled torques in the transmission (e.g., sliding friction, bearing damping, gear inefficiency, etc.). Moreover, for powered prostheses to mimic human joint impedance, they will need actuators that accurately render a wide range of mechanical impedances in a variety of ground contact conditions, including nearly free-swinging behavior in swing phase and stiff spring-like behavior in stance phase. For series-elastic prosthetic leg actuators, as in Open- Source Leg (OSL), these sudden output inertia changes present a challenge for traditional cascaded impedance control. In this paper we propose a solution based on disturbance observers and full state feedback (FSF) impedance control. With transmission disturbances attenuated, the FSF controller can use pole-zero placement to specify the actuator impedance that couples to the uncertain joint inertia. We validate our control framework on an OSL-like two-actuator dynamometry testbed.

Keywords: Human-Centered Robotics, Physically Assistive Devices, Motion Control
Abstract: We propose a new walking support robot concept, "Nimbus Guardian," designed to enhance the mobility of both healthy and frail elderly individuals who can walk independently. The proposed robot differs from traditional walker-type or cane-type aids by offering adaptive, minimal touch support based on the user's walking dynamics. Our goal is to realize versatile touch to the user as a preliminary study for developing the adaptive touch walking support robot. To achieve this, we've established a categorization system for walking support touch, outlining the specific types of assistance required for our robot. Based on these categorization, we've developed a prototype that improves the versatility of touch support (touch point, force, and initiator), adapting to the user's body. Our prototype is equipped to offer multiple touch support parts, adjusting to the user's physique. For versatile touch capabilities, we designed a motion control algorithm that includes a controller which directs the robot's wheel movements according to the chosen support points, and a state machine that provides multiple arm placements and movements. We have experimentally implemented this motion control algorithm in our prototype. Through experiments, we verified the touch versatility and discussed the prototype's utility and potential for further development.

Keywords: Prosthetics and Exoskeletons, Human Factors and Human-in-the-Loop, Force Control
Abstract: Partial hand amputations significantly affect the physical and psychosocial well-being of individuals, yet intuitive control of externally powered prostheses remains an open challenge. To address this gap, we developed a force-controlled prosthetic finger activated by electromyography (EMG) signals. The prototype, constructed around a wrist brace, functions as a supernumerary finger placed near the index, allowing for early-stage evaluation on unimpaired subjects. A neural network-based model was then implemented to estimate fingertip forces from EMG inputs, allowing for online adjustment of the prosthetic finger grip strength. The force estimation model was validated through experiments with ten participants, demonstrating its effectiveness in predicting forces. Additionally, online trials with four users wearing the prosthesis exhibited precise control over the device. Our findings highlight the potential of using EMG-based force estimation to enhance the functionality of prosthetic fingers.

Keywords: Intention Recognition, Behavior-Based Systems, Physically Assistive Devices
Abstract: Sit-to-stand and stand-to-sit motions are important in daily activities. However, elderly individuals often find these motions difficult to perform with declining lower limb strength, which causes a considerable reduction to their quality of life. In this study, a sensing method for controlling robotic assistive devices was proposed. This method utilizes electromyographic measurements and a deep neural network to predict motion initiation, and it estimates the timing of triggering assistive devices. Experimental results indicate that four muscle synergy patterns are required to represent the sit-to-stand and stand-to-sit motions together, with two of them being shared between both movements. Subsequently, a long short-term memory network was designed to forecast these two motions, and the result indicates that the prediction accuracy reached 92.95% +- 0.83% with forecasting time of 300 ms.

Keywords: Prosthetics and Exoskeletons, Medical Robots and Systems
Abstract: Amputations and limb loss can have detrimental effects on personal well-being. Although prosthetic devices can offer significant benefits helping amputees regain some of the lost dexterity, they often lack the required affordability and durability. Current affordable prosthetic designs have trended towards underactuation which leads to stable grasps but is often characterized by low durability. In this paper, a new chain driven, adaptive, underactuated finger design has been proposed for the development of affordable and highly durable prosthetic hands. The transmission mechanism employed is in the form of a steel roller chain. The finger phalanges are constructed of 3D printed PLA, and finger flexion is produced by pulling the internally routed roller chain that is rerouted using sprockets. In total, six 3D printed PLA sprockets are used for chain routing, with an emphasis on high force transmission. The performance of the proposed chain-driven finger was experimentally validated and compared with an analogous tendon-driven version. The metrics employed for this comparison were longevity, pinch grasp efficiency, force response, and maximum force capability. The chain driven finger was shown to have a higher maximum transmissible force, better long term durability, and no issues related to elongation (such as tendon elongation). The cost to manufacture the chain driven robotic finger is only 91 USD, making it an excellent solution for affordable prostheses.

Keywords: Prosthetics and Exoskeletons, Physically Assistive Devices, Wearable Robotics
Abstract: The online adaptation of exoskeleton control based on muscle activity sensing offers a promising approach to personalizing exoskeleton behavior based on the user’s biosignals. While electromyography (EMG)-based methods have demonstrated improvements in joint torque estimation, EMG sensors require direct skin contact and extensive post-processing. In contrast, force myography (FMG) measures normal forces resulting from changes in muscle volume due to muscle activity. We propose an FMG-based method to estimate knee and ankle joint torques by integrating joint angles and velocities with muscle activity data. We learn a model for joint torque estimation using Gaussian process regression (GPR). The effectiveness of the proposed FMG-based method is validated on isokinetic motions performed by ten participants. The model is compared to a baseline model that uses only joint angle and velocity, as well as a model augmented by EMG data. The results indicate that incorporating FMG into exoskeleton control can improve the estimation of joint torque for the ankle and knee joints in novel task characteristics within a single participant. Although the findings suggest that this approach may not improve the generalizability of estimates between multiple participants, they highlight the need for further research into its potential applications in exoskeleton control.

Keywords: Prosthetics and Exoskeletons
Abstract: The development of prosthetic feet that closely replicate the natural biomechanics of the human foot remains a significant challenge in prosthetics engineering. This paper presents the design and testing of a novel agonist-antagonist architecture for the ankle joint of a passive prosthetic foot featuring an adaptive sole. The ankle mechanism, inspired by the dynamics of the human leg-ankle-foot complex, utilizes compliant elements in an agonist-antagonist configuration to passively achieve an ankle torque close to that of a sound ankle without the need for external actuation. Concurrently, the adaptive sole adjusts its shape in response to different terrains, potentially improving stability and comfort for the user. The theoretical model underlying the proposed design is presented, followed by a preliminary validation through simulations. Finally, a prototype based on the new architecture is tested by a healthy subject using customized walking boots, demonstrating its potential to improve the functional performance of prosthetic feet in diverse environments.

Keywords: Intelligent Transportation Systems, Computer Vision for Transportation, Cooperating Robots
Abstract: Current LiDAR-based Vehicle-to-Everything (V2X) multi-agent perception systems have shown the significant success on 3D object detection. While these models perform well in the trained clean weather, they struggle in unseen adverse weather conditions with the domain gap. In this paper, we propose a Domain Generalization based approach, named V2X-DGW, for LiDAR-based 3D object detection on multi-agent perception system under adverse weather conditions. Our research aims to not only maintain favorable multi-agent performance in the clean weather but also promote the performance in the unseen adverse weather conditions by learning only on the clean weather data. To realize the Domain Generalization, we first introduce the Adaptive Weather Augmentation (AWA) to mimic the unseen adverse weather conditions, and then propose two alignments for generalizable representation learning: Trust-region Weather-invariant Alignment (TWA) and Agent-aware Contrastive Alignment (ACA). To evaluate this research, we add Fog, Rain, Snow conditions on two publicized multi-agent datasets based on physics-based models, resulting in two new datasets: OPV2V-w and V2XSet-w. Extensive experiments demonstrate that our V2X-DGW achieved significant improvements in the unseen adverse weathers.

Keywords: Automation Technologies for Smart Cities, Intelligent Transportation Systems, Autonomous Vehicle Navigation
Abstract: Many countries are developing an Urban Air Mobility (UAM) capability defining an Uncrewed Aircraft Systems (UAS) Traffic Management (UTM) architecture to allow safe UAS services in urban environments (e.g., delivery, inspection, air taxis, etc.). The main considerations are air worthiness, operator certification, air traffic management, C2 Link, detect and avoid (DAA), safety management, and security. In addition, if thousands of simultaneous UAS flights are to be achieved, it is not possible for them to be controlled individually by human operators. This makes it necessary to have a rigorous and safe automation methodology to handle such a number of flights. A lane-based airspace structure has been proposed which reduces the complexity of strategic deconfliction by providing UAS agents with a set of pre-defined airway corridors called lanes. This yields collateral benefits including UAS information privacy, robust contingency handling exploiting the lane structure, as well as improved observability and control of the air space. A robust set of UTM parameters and policies must be determined based on the performance characteristics of the deployed UAS platforms, and a methodology which constitutes a first step toward this end is proposed and demonstrated here. In order to realize this approach, a set of initial experiments have been performed to determine the constraints imposed by the UTM on UAS platform capabilities and vice versa. Initial implementation parameters and policies are defined. The major contribution here is a methodology to calibrate UTM safety parameters (e.g., headway, platform speed) in terms of specific platform models’ operational characteristics. That is, UTM parameters are a function of platform and not some arbitrarily imposed values. Safety uncertainty is then characterized by the calibration method.

Keywords: Intelligent Transportation Systems, Autonomous Vehicle Navigation
Abstract: Traffic sign detection plays an essential role in advanced driver assistance system (ADAS) or self-driving vehicles. Typically, deep neural networks are employed to analyze road scene images captured by an onboard camera. However, due to the significant variation in appearance of different traffic signs, the classification of high similarity patterns is still a challenging task. To address these issues, this paper presents an end-to-end traffic sign detection framework based on DETR. The proposed network incorporates data augmentation and negative sample learning to mitigate the problem of data imbalance and enhance the model recognition capability effectively. An UASPP module (Upsample Atrous Pyramid Pooling) is introduced to integrate multi-scale features and global information. In the experiments, the performance evaluation has demonstrated the improvement of mAP by 3.9% on TT100K and 36.3% on GTSDB compared to state-of-the-art methods. The code and datasets are available at https://github.com/chinglun/TS-DETR.

Keywords: Automation Technologies for Smart Cities, Building Automation, Energy and Environment-Aware Automation
Abstract: This paper introduces an innovative approach to designing a user-based Heating, Ventilation, and Air- Conditioning (HVAC) system management connected with the District Energy Management System. By classifying the users into dynamic energy consumption classes to reward energy efficiency and penalize excessive use, users can modify their behavior to pass to a less expensive and more virtuous consumption class. To this aim, a blockchain platform determines the rewards and penalties and, by a K-means clustering algorithm, categorizes users into respective groups. Then, a Class Follower Problem is formulated and solved by a Model Predictive Control (MPC) strategy integrated with a Long Short-Term Memory network as a predictive model. If the users follow the suggestions proposed by the controller, i.e., the thermostat set-points and the time intervals in which the HVAC system must be switched off or on, the users can be located in a more virtuous consumption class. A case study conducted within an energy district in Bari (Italy) shows how the proposed architectural framework tuned thermal regulation in intelligent buildings while concurrently achieving energy optimization

Keywords: Localization, Autonomous Agents, Intelligent Transportation Systems
Abstract: In this paper, we propose a non-parametric method for estimating the posterior distribution of GNSS integer ambiguity. Is is difficult to estimate the posterior probability of discrete integer ambiguities directly from carrier phase observations due to the unclear domain definition. We thus introduce the positional likelihood field that accumulates the ambiguity function method values in the positional space and then estimate the integer ambiguity distributions by marginalizing the likelihood over the entire position. Because the positional likelihood field is defined in the positional space, it enables ease of carrier phase likelihood accumulation. In order to correctly estimate the posterior distribution, however, a sufficient density of the samples is required that results in a large computational cost. The proposed method enables large-scale sampling by taking advantage of GPU parallel processing. Experimental results demonstrate that the proposed method enables accurate and robust estimation of integer ambiguity distributions, contributing to improved centimeter-level position estimation accuracy. In addition, the histograms provided quantitative evidence of events in urban environments where integer ambiguity is not uniquely determined.

Keywords: Automation Technologies for Smart Cities, Modeling and Simulating Humans, Simulation and Animation
Abstract: Realistic crowd simulations are essential for immersive virtual environments, relying on both individual behaviors (microscopic dynamics) and overall crowd patterns (macroscopic characteristics). While recent data-driven methods like deep reinforcement learning improve microscopic realism, they often overlook critical macroscopic features such as crowd density and flow, which are governed by spatio-temporal spawn dynamics, namely, when and where agents enter a scene. Traditional methods, like random spawn rates, stochastic processes, or fixed schedules, are not guaranteed to capture the underlying complexity or lack diversity and realism. To address this issue, we propose a novel approach called nTPP-GMM that models spatio-temporal spawn dynamics using Neural Temporal Point Processes (nTPPs) that are coupled with a spawn-conditional Gaussian Mixture Model (GMM) for agent spawn and goal positions. We evaluate our approach by orchestrating crowd simulations of three diverse real-world datasets with nTPP-GMM. Our experiments demonstrate the orchestration with nTPP-GMM leads to realistic simulations that reflect real-world crowd scenarios and allow crowd analysis.

Keywords: Autonomous Vehicle Navigation, Integrated Planning and Learning, Intelligent Transportation Systems
Abstract: We introduce RMP-YOLO, a unified framework designed to provide robust motion predictions even with incomplete input data. Our key insight stems from the observation that complete and reliable historical trajectory data plays a pivotal role in ensuring accurate motion prediction. Therefore, we propose a new paradigm that prioritizes the reconstruction of intact historical trajectories before feeding them into the prediction modules. Our approach introduces a novel scene tokenization module to enhance the extraction and fusion of spatial and temporal features. Following this, our proposed recovery module reconstructs agents' incomplete historical trajectories by leveraging local map topology and interactions with nearby agents. The reconstructed, clean historical data is then integrated into the downstream prediction modules. Our framework is able to effectively handle missing data of varying lengths and remains robust against observation noise, while maintaining high prediction accuracy. Furthermore, our recovery module is compatible with existing prediction models, ensuring seamless integration. Extensive experiments validate the effectiveness of our approach, and deployment in real-world autonomous vehicles confirms its practical utility. In the 2024 Waymo Motion Prediction Competition, our method, RMP-YOLO, achieves state-of-the-art performance, securing third place. Our code is open-source at https://github.com/ggosjw/RMP-YOLO.

Keywords: Sensor Fusion, Mapping, Prosthetics and Exoskeletons
Abstract: For leg exoskeletons to operate effectively in real-world environments, they must be able to perceive and understand the terrain around them. However, unlike other legged robots, exoskeletons face specific constraints on where depth sensors can be mounted due to the presence of a human user. These constraints lead to a limited Field Of View (FOV) and greater sensor motion, making odometry particularly challenging. To address this, we propose a novel odometry algorithm that integrates proprioceptive data from the exoskeleton with point clouds from a depth camera to produce accurate elevation maps despite these limitations. Our method builds on an extended Kalman filter (EKF) to fuse kinematic and inertial measurements, while incorporating a tailored iterative closest point (ICP) algorithm to register new point clouds with the elevation map. Experimental validation with a leg exoskeleton demonstrates that our approach reduces drift and enhances the quality of elevation maps compared to a purely proprioceptive baseline, while also outperforming a more traditional point cloud map-based variant.

Keywords: Localization
Abstract: Monocular visual inertial odometry (VIO) has facilitated a wide range of real-time motion tracking applications, thanks to the small size of the sensor suite and low power consumption. To successfully bootstrap VIO algorithms, the initialization module is extremely important. Most initialization methods rely on the reconstruction of 3D visual point clouds. These methods suffer from high computational cost as state vector contains both motion states and 3D feature points. To address this issue, some researchers recently proposed a structureless initialization method, which can solve the initial state without recovering 3D structure. However, this method potentially compromises performance due to the decoupled estimation of rotation and translation, as well as linear constraints. To improve its accuracy, we propose novel structureless visual-inertial bundle adjustment to further refine previous structureless solution. Extensive experiments on real-world datasets show our method significantly improves the VIO initialization accuracy, while maintaining real-time performance.

Keywords: Deep Learning for Visual Perception, Computer Vision for Automation, SLAM
Abstract: Deep visual odometry, despite extensive research, still faces limitations in accuracy and generalizability that prevent its broader application. To address these challenges, we propose an Oriented FAST and Rotated BRIEF (ORB)-guided visual odometry with selective online adaptation named ORB-SfMLearner. We present a novel use of ORB features for learning-based ego-motion estimation, leading to more robust and accurate results. We also introduce the cross-attention mechanism to enhance the explainability of PoseNet and have revealed that driving direction of the vehicle can be explained through the attention weights. To improve generalizability, our selective online adaptation allows the network to rapidly and selectively adjust to the optimal parameters across different domains. Experimental results on KITTI and vKITTI datasets show that our method outperforms previous state-of-the-art deep visual odometry methods in terms of ego-motion accuracy and generalizability.

Keywords: Visual-Inertial SLAM, Localization, SLAM
Abstract: Energy-efficient visual-inertial motion tracking on SWAP-constrained edge devices (e.g., drones and AR glasses) is essential but challenging. Our previous work introduced the first-of-its-kind quantized visual-inertial odometry (QVIO), utilizing either raw measurement quantization (zQVIO) or single-bit residual quantization (rQVIO). While QVIO has demonstrated significant data transfer reduction with competitive performance, but it has limitations. Specifically, zQVIO directly quantizes raw measurements into multi-bit values, while requiring the ad-hoc inflation of measurement noise to account for quantization errors. On the other hand, rQVIO is limited to single-bit measurement with a certain accuracy loss. This work introduces QVIO2 to address these issues. The proposed QVIO2 improves data quantization strategies and derives a Maximum A Posteriori (MAP) quantized estimator that rigorously handles both multi-bit and single-bit, raw and residual quantized measurements in a unified manner. These improvements lead to more communication-efficient and accurate systems. Additionally, we optimize the communication protocol to further reduce data transfer by eliminating un- necessary transmissions. Extensive numerical and experimental results demonstrate reduced communication requirements and improved accuracy. Compared to the previous QVIO system, zQVIO2 achieves the same accuracy with a 30% reduction in data transfer, while rQVIO2 improves accuracy without increasing data communication. In real-world scenarios, our new zQVIO2 and rQVIO2 have demonstrated nearly no accuracy loss with only 4.6 bits and 3.5 bits of data communication, achieving compression rates of 7× and 9.1×.

Keywords: Visual-Inertial SLAM, Sensor Fusion, Localization
Abstract: Visual-inertial odometry (VIO), which fuses noisy inertial readings and camera measurements to provide 3D motion tracking, is a foundational component in many autonomous applications. With the increasing use of next-generation edge devices (e.g., IoT devices, nano drones, and mobile robotics) that are constrained by limited power, resources, and multi-tasking demands, balancing computational efficiency and accuracy in VIO estimators has become more critical than ever. Historically, state estimation algorithms have been developed using either optimization-based or filtering-based methods, with the key distinction being the ability to relinearize measurements and correct state estimates iteratively. While it has been widely claimed that iterative methods improve accuracy by allowing for the reduction of error through relinearization at a higher computational demand. Conversely, filtering methods are more efficient but may suffer from significant linearization errors. However, these trade-offs have not been thoroughly examined in the context of visual-inertial motion tracking. In this paper, we conduct the first comprehensive study on the impact of iterative algorithms in sliding-window VIO. We analyze the relinearization of IMU and camera measurements separately, providing insights into how each affects system performance. By considering key factors such as system observability and measurement processes, we offer a deeper understanding of VIO estimator behavior. Our findings, supported by proof-of-concept real-world tests, provide practical guidelines for balancing accuracy and efficiency, helping practitioners determine when to prioritize iterative methods or simpler filtering approaches while encouraging researchers and engineers to rethink VIO design for optimal resource allocation.

Keywords: SLAM, Visual-Inertial SLAM
Abstract: Terrestrial-aerial bimodal vehicles (TABVs) have attracted great attention because of their advantages over single-model robots. TABVs can provide superior obstacle avoidance capability (flying mode) and safe mobility with long duration (ground mode), offering enhanced adaptability and flexibility in various challenging environments. However, a robust localization approach becomes the bottleneck to stably applying the TABVs in real-world tasks. In this paper, we present an environment-aware robust localization system specifically designed for passive-wheel-based TABVs, which feature two passive wheels alongside a standard quadrotor. The localization system tightly integrates data from multiple sensors, including a camera, Inertial Measurement Units (IMUs), encoders, and single-point laser distance sensors. First, we introduce a terrain-aware odometer model that accurately estimates terrain slope and vehicle's velocity by fusing gyroscope, encoder, and single-point laser measurements. Then, we propose an anomaly-aware method that senses anomalous sensors and dynamically adjusts the optimization weights accordingly. By explicitly estimating the environmental conditions, such as ground terrain slopes and visual information qualities, the robot can achieve accurate and robust localization results on the ground. To validate our localization approach, we conducted extensive experiments across various challenging scenarios, demonstrating the effectiveness and reliability of our system for real-world applications.

Keywords: Visual-Inertial SLAM, Sensor Fusion, SLAM
Abstract: Despite extensive research in robust visual-inertial navigation systems(VINS) in dynamic environments, many approaches remain vulnerable to objects that suddenly start moving, which are referred to as abruptly dynamic objects. In addition, most approaches have considered the effect of dynamic objects only at the feature association level. In this study, we observed that the state estimation diverges when errors from false correspondences owing to moving objects incorrectly propagate into the IMU bias terms. To overcome these problems, we propose a robust VINS framework called DynaVINS++, which employs a) adaptive truncated least square method that adaptively adjusts the truncation range using both feature association and IMU preintegration to effectively minimize the effect of the dynamic objects while reducing the computational cost, and b) stable state recovery with bias consistency check to correct misestimated IMU bias and to prevent the divergence caused by abruptly dynamic objects. As verified in both public and real-world datasets, our approach shows promising performance in dynamic environments, including scenes with abruptly dynamic objects.

Keywords: Telerobotics and Teleoperation, Haptics and Haptic Interfaces, Human Factors and Human-in-the-Loop
Abstract: Force feedback generally increases the effectiveness of execution and sense of embodiment on telemanipulation systems. However, systems with force feedback are vulnerable to time delays, reducing their transparency and stability. In this paper, we implement a bi-directional impedance reflection controller, a concept that was presented already in 1989 by Blake Hannaford [1] but was never fully implemented. In this method, the simplified impedances of the operator and the environment are estimated and reflected back to the remote robot and haptic interface, respectively. A trajectory predictor is added to compensate for the delayed motion. We then evaluated the effectiveness of the system in a user study, comparing it to a system with a classical bilateral impedance controller with passivity layers. Three time delay groups (0, 10, and 20 ms one-way delay) of 10 participants each executed different tasks with both controllers. The results show that the bi-directional impedance reflection controller performs significantly better in the 10 ms and 20 ms time delay groups, in terms of task performance, user experience and sense of embodiment. We conclude that this study is the first to show that bi-directional impedance reflection is robust to time delays of at least 20 ms.

Keywords: Deep Learning for Visual Perception, Gesture, Posture and Facial Expressions, Human Detection and Tracking
Abstract: In the realm of robotics, teleoperation plays a pivotal role in performing high-risk or intricate tasks, and obtaining precise 3D whole-body pose is crucial for this purpose. Traditional two-stage methods have limitations in estimating different body parts, leading to complex systems and higher estimation errors. In order to address these issues,the paper introduces a novel framework called Graph High-Resolution Network(GraphHRNet) for accurate 3D whole-body pose estimation, which is essential for the teleoperation of humanoid robots. GraphHRNet effectively captures global structural information and local details by integrating a High-Resolution Module and a Multi-branch Regression Module. The High-Resolution Module utilizes an enhanced graph convolution kernel to fuse multi-scale features, capturing global information, while the Multi-branch Regression Module focuses on refining and predicting accurate 3D coordinates for intricate body parts such as hands and face. Experimental results on the H3WB dataset demonstrate that GraphHRNet surpasses state-of-the-art(SOTA) methods in 3D whole-body pose estimation , significantly improving performance. Furthermore, the paper explores the potential application of this approach in a teleoperation system for humanoid robots, providing an intuitive and high-fidelity solution for remotely executing complex tasks. The code have been publicly available at https://github.com/Z-mingyu/GraphHRNet.git

Keywords: Field Robots, Telerobotics and Teleoperation, Deep Learning for Visual Perception
Abstract: Teleoperation is an important technology to enable supervisors to control agricultural robots remotely. However, environmental factors in dense crop rows and limitations in network infrastructure hinder the reliability of data streamed to teleoperators. These issues result in delayed and variable frame rate video feeds that often deviate significantly from the robot's actual viewpoint. We propose a modular learning-based vision pipeline to generate delay-compensated images in real-time for supervisors. Our extensive offline evaluations demonstrate that our method generates more accurate images compared to state-of-the-art approaches in our setting. Additionally, ours is one of the few works to evaluate a delay-compensation method in outdoor field environments with complex terrain on data from a real robot in real-time. Resulting videos and code are provided at https://sites.google.com/illinois.edu/comp-teleop.

Keywords: Imitation Learning, Force Control, Deep Learning in Grasping and Manipulation
Abstract: In most contact-rich manipulation tasks, humans apply time-varying forces to the target object, compensating for inaccuracies in the vision-guided hand trajectory. However, current robot learning algorithms primarily focus on trajectory-based policy, with limited attention given to learning force-related skills. To address this limitation, we introduce ForceMimic, a force-centric robot learning system, providing a natural, force-aware and robot-free robotic demonstration collection system, along with a hybrid force-motion imitation learning algorithm for robust contact-rich manipulation. Using the proposed ForceCapture system, an operator can peel a zucchini in 5 minutes, while force-feedback teleoperation takes over 13 minutes and struggles with task completion. With the collected data, we propose HybridIL to train a force-centric imitation learning model, equipped with hybrid force-position control primitive to fit the predicted wrench-position parameters during robot execution. Experiments demonstrate that our approach enables the model to learn a more robust policy under the contact-rich task of vegetable peeling, increasing the success rates by 54.5% relatively compared to state-of-the-art pure-vision-based imitation learning. Hardware, code, data and more results can be found on the project website at https://forcemimic.github.io.

Keywords: Imitation Learning, Learning from Demonstration, Data Sets for Robot Learning
Abstract: Imitation learning is a promising approach for learning robot policies with user-provided data. The way demonstrations are provided, i.e., demonstration modality, influences the quality of the data. While existing research shows that kinesthetic teaching (physically guiding the robot) is preferred by users for the intuitiveness and ease of use, the majority of existing manipulation datasets were collected through teleoperation via a VR controller or spacemouse. In this work, we investigate how different demonstration modalities impact downstream learning performance as well as user experience. Specifically, we compare low-cost demonstration modalities including kinesthetic teaching, teleoperation with a VR controller, and teleoperation with a spacemouse controller. We experiment with three table-top manipulation tasks with different motion constraints. We evaluate and compare imitation learning performance using data from different demonstration modalities, and collected subjective feedback on user experience. Our results show that kinesthetic teaching is rated the most intuitive for controlling the robot and provides cleanest data for best downstream learning performance. However, it is not preferred as the way for large-scale data collection due to the physical load. Based on such insight, we propose a simple data collection scheme that relies on a small number of kinesthetic demonstrations mixed with data collected through teleoperation to achieve the best overall learning performance while maintaining low data-collection effort.

Keywords: Telerobotics and Teleoperation, Cognitive Human-Robot Interaction, Human Factors and Human-in-the-Loop, Human-Centered Robotics
Abstract: Advances in robot teleoperation have enabled groundbreaking innovations in many fields, such as space exploration, healthcare, and disaster relief. The human operator's performance plays a key role in the success of any teleoperation task, with prior evidence suggesting that operator stress and workload can impact task performance. As robot teleoperation is currently deployed in safety-critical domains, it is essential to analyze how different stress and workload levels impact the operator. We are unaware of any prior work investigating how both stress and workload impact teleoperation performance. We conducted a novel study (n=24) to jointly manipulate users' stress and workload and analyze the user's performance through objective and subjective measures. Our results indicate that, as stress increased, over 70% of our participants performed better up to a moderate level of stress; yet, the majority of participants performed worse as the workload increased. Importantly, our experimental design elucidated that stress and workload have related yet distinct impacts on task performance, with workload mediating the effects of distress on performance (p<.05).

Keywords: Telerobotics and Teleoperation, Human Performance Augmentation, Virtual Reality and Interfaces
Abstract: In recent years, human-robot interaction (HRI) has become an increasingly important field of research. The human experience during HRI tasks like teleoperation or turn-taking largely depends on the interface design between the robot and the user. Designing an intuitive user interface (UI) between an arbitrary M-dimensional input device and an N-degree of freedom (DOF) robot remains a significant challenge. This paper proposes a novel UI design approach named the Parallel Neural Networks Adaptive User Interface (PNNUI). PNNUI utilizes two parallel neural networks to learn and then improve the teleoperation performance of users by minimizing task completion time and maximizing motion smoothness. Our method is designed to learn an unintuitive input-output map between user interface hardware and the robot by minimizing task completion time in an offline unsupervised learning scheme based on Neural Networks (NNs) and Genetic Algorithms. Secondly, PNNUI minimizes teleoperation jerk online by adapting the weights of a parallel neural network. We experimentally evaluated the resulting UI for teleoperating a 3-DOF nonholonomic robot through a conventional joystick with three inputs. Twenty human subjects operated the robot along an obstacle course in several conditions. The statistical analysis of the user trial data shows that PNNUI improves the human experience in robot teleoperation by maximizing smoothness while maintaining the completion time of the offline learning scheme. Furthermore, the abstract nature of our formulation enables the customization of performance measures, which extends its applicability to other interface devices and HRI tasks, particularly those that are not intuitive to start with.

Keywords: Reinforcement Learning, Imitation Learning, Deep Learning in Grasping and Manipulation
Abstract: In this work, we focus on addressing the long-horizon packing tasks in densely cluttered scenes. Such tasks require policies to effectively manage severe occlusions among objects and continually produce precise actions based on visual observations. We propose a vision-based Hierarchical policy for Cluttered-scene Long-horizon Manipulation (HCLM). It employs a high-level policy and three options to select and instantiate three parameterized action primitives: push, pick, and place. We first train the two-stream pick and place options by behavior cloning (BC). Subsequently, we use hierarchical reinforcement learning (HRL) to train the high-level policy and push option. During HRL, we propose a Spatially Extended Q-update (SEQ) to augment the updates for the push option and a Two-Stage Update Scheme (TSUS) to alleviate the non-stationary transition problem in updating the high-level policy. We demonstrate that HCLM significantly outperforms baseline methods in terms of success rate and efficiency in diverse tasks both in simulation and real world. The ablation studies also validate the key roles of SEQ and TSUS in HRL.

Keywords: Reinforcement Learning, Agent-Based Systems, Autonomous Agents
Abstract: Multi-agent systems in non-stationary environments face challenges due to rapidly changing dynamics, leading to quick obsolescence of experiences in the replay buffer. To address this, we propose the Adaptive Experience Replay with Attention-Based Sequence Embedding (AERAS) framework, which integrates sequence embedding with an attention mechanism to prioritize experiences based on their relevance. By assigning adaptive weights, AERAS emphasizes relevant experiences while diminishing the impact of outdated ones, enhancing efficiency and learning performance in multi-agent reinforcement learning. Evaluations on the StarCraft II Multi-Agent Challenge and Google Research Football environments show that AERAS consistently outperforms state-of-the-art methods, achieving faster convergence and higher win rates. Ablation studies confirm the essential roles of sequence embedding and attention mechanisms in boosting AERAS's robustness and adaptability, underscoring its effectiveness in managing non-stationary environments within multi-agent systems.

Keywords: Reinforcement Learning, Human Factors and Human-in-the-Loop, Deep Learning Methods
Abstract: Preference-Based reinforcement learning (PBRL) learns directly from the preferences of human teachers regarding agent behaviors without needing meticulously designed reward functions. However, existing PBRL methods often learn primarily from explicit preferences, neglecting the possibility that teachers may choose equal preferences. This neglect may hinder the understanding of the agent regarding the task perspective of the teacher, leading to the loss of important information. To address this issue, we introduce the Equal Preference Learning Task, which optimizes the neural network by promoting similar reward predictions when the behaviors of two agents are labeled as equal preferences. Building on this task, we propose a novel PBRL method, Multi-Type Preference Learning (MTPL), which allows simultaneous learning from equal preferences while leveraging existing methods for learning from explicit preferences. To validate our approach, we design experiments applying MTPL to four existing state-of-the-art baselines across ten locomotion and robotic manipulation tasks in the DeepMind Control Suite. The experimental results indicate that simultaneous learning from both equal and explicit preferences enables the PBRL method to more comprehensively understand the feedback from teachers, thereby enhancing feedback efficiency. Project page: url{https://github.com/FeiCuiLengMMbb/paper_MTPL}