Keywords: Field Robots, Legged Robots, Industrial Robots
Abstract: Quadruped robots are proliferating in industrial environments where they carry sensor payloads and serve as autonomous inspection platforms. Despite the advantages of legged robots over their wheeled counterparts on rough and uneven terrain, they are still unable to reliably negotiate a ubiquitous feature of industrial infrastructure: ladders. Inability to traverse ladders prevents quadrupeds from inspecting dangerous locations, puts humans in harm's way, and reduces industrial site productivity. In this paper, we learn quadrupedal ladder climbing via a reinforcement learning-based control policy and a complementary hooked end effector. We evaluate the robustness in simulation across different ladder inclinations, rung geometries, and inter-rung spacings. On hardware, we demonstrate zero-shot transfer with an overall 90% success rate at ladder angles ranging from 70° to 90°, consistent climbing performance during unmodeled perturbations, and climbing speeds 232x faster than the state of the art. This work expands the scope of industrial quadruped robot applications beyond inspection on nominal terrains to challenging infrastructural features in the environment, highlighting synergies between robot morphology and control policy when performing complex skills. More information can be found at the project website: https://sites.google.com/leggedrobotics.com/climbingladders .

Keywords: Climbing Robots, Legged Robots, Flexible Robotics
Abstract: This paper introduces FlipWalker, a novel underactuated robot locomotion system inspired by Jacob's Ladder illusion toy, designed to traverse challenging terrains where wheeled robots often struggle. Like the Jacob’s Ladder toy, FlipWalker features two interconnected segments joined by flexible cables, enabling it to pivot and flip around singularities in a manner reminiscent of the toy’s cascading motion. Actuation is provided by motor-driven legs within each segment that push off either the ground or the opposing segment, depending on the robot’s current configuration. A physics-based model of the underactuated flipping dynamics is formulated to elucidate the critical design parameters governing forward motion and obstacle clearance or climbing. The untethered prototype weighs 0.78 kg, achieves a maximum flipping speed of 0.2 body lengths per second. Experimental trials on artificial grass, river rocks, and snow demonstrate that FlipWalker’s flipping strategy, which relies on ground reaction forces applied normal to the surface, offers a promising alternative to traditional locomotion for navigating irregular outdoor terrain.

Keywords: Gesture, Posture and Facial Expressions, Emotional Robotics, Perception-Action Coupling
Abstract: Accurate pain expression synthesis is essential for improving clinical training and human-robot interaction. Current Robotic Patient Simulators (RPSs) lack realistic pain facial expressions, limiting their effectiveness in medical training. In this work, we introduce PainDiffusion, a generative model that synthesizes naturalistic facial pain expressions. Unlike traditional heuristic or autoregressive methods, PainDiffusion operates in a continuous latent space, ensuring smoother and more natural facial motion while supporting indefinite-length generation via diffusion forcing. Our approach incorporates intrinsic characteristics such as pain expressiveness and emotion, allowing for personalized and controllable pain expression synthesis. We train and evaluate our model using the BioVid HeatPain Database. Additionally, we integrate PainDiffusion into a robotic system to assess its applicability in real-time rehabilitation exercises. Qualitative studies with clinicians reveal that PainDiffusion produces realistic pain expressions, with a 31.2% ± 4.8% preference rate against ground-truth recordings. Our results suggest that PainDiffusion can serve as a viable alternative to real patients in clinical training and simulation, bridging the gap between synthetic and naturalistic pain expression.

Keywords: Social HRI, Humanoid Robot Systems, Imitation Learning
Abstract: Nonverbal communication plays a crucial role in both human-human and human-robot interactions (HRIs), where facial expressions convey emotions, intentions and trust. Enabling humanoid robots to generate human-like facial reactions in response to human speech and facial behaviours remains significant challenges. In this work, we leverage human-human interaction (HHI) datasets to train a humanoid robot, allowing it to learn and imitate facial reactions to both speech and facial expression inputs. Specifically, we extend a sequence-to-sequence (Seq2Seq)-based framework that enables robots to simulate human-like virtual facial expressions that are appropriate for responding to the perceived human user behaviours. Then, we propose a deep neural network-based motor mapping model to translate these expressions into physical robot movements. Experiments demonstrate that our facial reaction–motor mapping framework successfully enables robotic self-reactions to various human behaviours, where our model can best predict 50 frames (two seconds) of facial reactions in response to the input user behaviour of the same duration, aligning with human cognitive and neuromuscular processes. Our code is provided at https://github.com/mrsgzg/Robot_Face_Reaction.

Keywords: Field Robots, Marine Robotics, Surveillance Robotic Systems
Abstract: Monitoring of waterways such as remote and hazardous rivers and streams is important so as to assess the impact of external factors including construction runoff or climate change. Versatile, autonomous robotic boats can offer excellent environmental inspection and monitoring solutions for remote, dangerous, or access protected water bodies but they have several shortcomings in terms of maneuverability. This paper proposes an environmental inspection system consisting of an autonomous data collection buoy which is designed to be deployed to inaccessible river systems using a drone. The system can perform a drop off and pickup of the buoy depending on the requirements of a particular location and monitoring task. Utilising the natural flow of the river the buoy autonomously steers down, using GPS and magnetometers so as to maintain the desired trajectory. The buoy is capable of measuring water temperature but it can also be equipped with a range of sensors such as water oxygen meter, sonar for river bed inspection, or turbidity for water clarity. This paper describes the system design, presents an analysis of the self-righting capabilities of the buoy, and shows a full system demonstration at the Orewa River in Auckland, New Zealand.

Keywords: Telerobotics and Teleoperation, Robotics in Hazardous Fields, Search and Rescue Robots
Abstract: Mobile manipulators integrate the locomotion flexibility of quadruped robots with the operational capabilities of robotic manipulators. This integrated system is particularly effective for teleoperating explosive ordnance disposal (EOD) tasks in hazardous environments, enabling the safe handling of explosive devices. However, when the quadruped operates in narrow corridors or cluttered spaces, its ability to reposition is limited. This limitation, combined with targets located laterally relative to the robot, poses critical challenges for achieving rapid and intuitive teleoperation of the manipulator. Existing manipulator mapping methods either fail to support lateral teleoperation or lack proper coordinate transformations, leading to mismatches between the intended and actual movement directions of the leader and follower devices. This reduces operational intuitiveness and increases the cognitive load on human operators. To overcome these issues, we propose a hybrid mapping method that combines joint-space velocity control with Cartesian-space control. This method leverages joint-space velocity commands for rapid manipulator reorientation, while employing Cartesian space commands to achieve precise end-effector teleoperation. Furthermore, we introduce a virtual base coordinate frame that adaptively adjusts in response to the manipulator's reorientation. This adaptive compensation ensures that the visual feedback from the camera mounted on the end-effector remains consistent and intuitive. The proposed method was validated through experiments on a quadruped robot equipped with a manipulator in an EOD scenario. Results demonstrated significant improvements, including a 100% success rate, 43.9% task duration reduction, and 31.7% NASA-TLX score decrease, indicating decreased cognitive load and enhanced task efficiency compared to baseline methods.

Keywords: Object Detection, Segmentation and Categorization, Computer Vision for Transportation, Recognition
Abstract: Obstacles on railroads significantly increase the risk of traveling with a lot of train accidents caused by undetected obstacles. The obstacles disturb both the shipments of goods and the transportation of people leading to delays and damage which then result in substantial financial losses. Following natural disasters, manually locating and removing obstacles is not only time-consuming but also hazardous for the personnel involved. To address these challenges, this paper proposes an object detection system that can be implemented on an aerial drone to detect obstacles on the railway. This approach aims to enhance railway safety, reduce costs, and ensure the timely delivery of essential goods such as food and medical supplies during emergencies.

Keywords: Reinforcement Learning, AI-Enabled Robotics, Motion Control
Abstract: Articulated tracked robots face significant challenges in maintaining stable locomotion over uneven terrain due to unknown contact points between tracks and ground, which are critical for dynamic control. Unlike legged robots where contact locations can be predicted, tracked systems require real-time adaptation to varying terrains. This paper presents C-TRAC, a terrain-adaptive control framework that integrates reinforcement learning with a contact-modeling variational autoencoder (C-VAE) to enable robust obstacle traversal. We first train a C-VAE in simulation to reconstruct high-fidelity contact information (position and binary probability) from noisy sensor measurements. This model learns a latent representation of terrain contacts, capturing complex interactions between the robot's kinematics and environment. Subsequently, we employ an asymmetric Soft Actor-Critic (SAC) algorithm to optimize a control policy that leverages the predicted contact data for adaptive track control during locomotion. Extensive experiments validate C-TRAC in both simulated and real-world scenarios. In benchmark tests against state-of-the-art (SOTA) methods using RoboCup Rescue Robot League environments, our approach achieves superior obstacle traversal speed (up to 66.67% faster on 45^{circ} staircase) and stability (up to 47.53% more stable on the oblique terrace) compared to contact-agnostic RL baselines and model-based methods. Notably, zero-shot sim-to-real transfer demonstrates consistent performance in unstructured outdoor ruins, also confirming the framework's practicality.

Keywords: Mobile Manipulation, Modeling, Control, and Learning for Soft Robots, Planning, Scheduling and Coordination
Abstract: This paper introduces the lacrosse mobile manipulator, a robotic system designed to play lacrosse. We focus on the task of ball passing between two robots, where challenges exist, including managing a small ball in the soft lacrosse head and interacting with a fast-moving ball. In this study, we develop innovative neural-network based learning approaches to enhance performance in dynamic environments. The robots are autonomously controlled in a decentralized manner. A combination of analytical physical-based and machine learning methods is employed to refine motion planner and ball landing predictions in both the throwing and catching processes. The system achieves satisfactory performance in real-world ball-passing experiments.

Keywords: Model Learning for Control, Modeling, Control, and Learning for Soft Robots
Abstract: Cable-driven soft manipulators, with inherent compliance and hyper-redundancy, offer significant advantages in unstructured environments but present formidable challenges in modeling of inverse kinematics due to nonlinear deformations and underactuation. In this paper, building on a modified forward kinematic model, a physics-informed neural networks (PINN) framework based on spatiotemporal data is proposed for efficient inverse kinematics computation of cable-driven soft robotic manipulators. A geometrically exact forward kinematic model is constructed under the Piecewise Constant Curvature (PCC) assumption, extended to multi-section configurations, and enhanced by cable deflection compensation to account for practical routing constraints. Experimental validation shows a 40.11% reduction in end-effector positioning error (average 15.98 mm) when deflection effects are included. The proposed PINN architecture takes time and section count as inputs and outputs the corresponding manipulator configuration, enabling unified spatiotemporal trajectory tracking by minimizing elastic energy while satisfying kinematic constraints. Compared to particle swarm optimization (PSO), which requires iterative computation for each trajectory sample, the proposed method reduces computational time by over 71.9%, demonstrating superior efficiency in solving redundant inverse kinematics problems. This work bridges data-driven and mechanics-based approaches, offering a scalable solution for real-time control of soft manipulators.

Keywords: Modeling, Control, and Learning for Soft Robots, Dynamics, Simulation and Animation
Abstract: Flexible tools such as golf clubs and sports prostheses used for exercise are generally constructed from materials that are both strong and lightweight enough to withstand the weight and speed of human movement. The authors have previously proposed a hybrid-link system that integrates a rigid-link system with a flexible structure, achieving forward dynamics simulations of the hybrid-link system with floating base-link, such as a human and a humanoid. However, numerical simulations of models with stiff and light flexible structures diverge and are difficult to realize. Using implicit integration as a forward dynamics calculation method is expected to improve the stability of the calculation. However, it requires information on the gradient of the equations of motion． Therefore, this study extends the comprehensive dynamic gradient calculation method proposed for rigid-link systems to hybrid-link systems with floating base-link; moreover, a forward dynamics simulation with contact force calculation is realized using implicit integration in a robot arm and a humanoid with flexible structure.

Keywords: Modeling, Control, and Learning for Soft Robots, AI-Based Methods, Tendon/Wire Mechanism
Abstract: Cable-driven serpentine manipulators hold great potential in unstructured environments, offering obstacle avoidance, multi-directional force application, and a lightweight design. By placing all motors and sensors at the base and employing plastic links, we can further reduce the arm's weight. To demonstrate this concept, we developed a 9-degree-of-freedom cable-driven serpentine manipulator with an arm length of 545 mm and a total mass of only 308 g. However, this design introduces flexibility-induced variations, such as cable slack, elongation, and link deformation. These variations result in discrepancies between analytical predictions and actual link positions, making pose estimation more challenging. To address this challenge, we propose a physical reservoir computing based pose estimation method that exploits the manipulator's intrinsic nonlinear dynamics as a high-dimensional reservoir. Experimental results show a mean pose error of 4.3 mm using our method, compared to 4.4 mm with a baseline long short-term memory network and 39.5 mm with an analytical approach. This work provides a new direction for control and perception strategies in lightweight cable-driven serpentine manipulators leveraging their intrinsic dynamics.

Keywords: Modeling, Control, and Learning for Soft Robots, Soft Robot Applications
Abstract: The integration of origami structures into soft robotics has enriched the adaptability and functionality of the soft robots. Our research group has developed a cable-driven origami robot attached to an arc frame, which enables its deployment in an MR bore and manipulation of medical tools. However, the control of such origami robots still faces challenges such as nonlinear dynamics, unstructured environment, and external payload. This paper introduces an active modeling compensation control method using Koopman operator (K-AMCC) for the Yoshimura origami manipulator, enabling its accurate trajectory tracking with payloads under different orientations. This active modeling method exploits Koopman operator theory and Kalman filter to estimate the model error and synergies the linear quadratic regulator to compensate the modeling errors. The rectangular and circular trajectory tracking experiments under varying payloads and orientations were carried out. The results demonstrate the K-AMCC method's ability to improve trajectory accuracy significantly, which lays a solid foundation for further medical applications such as needle manipulation and laser ablation in an MR environment.

Keywords: Modeling, Control, and Learning for Soft Robots, Localization, Sensor Fusion
Abstract: Tensegrity robots, characterized by a synergistic assembly of rigid rods and elastic cables, form robust structures that are resistant to impacts. However, this design introduces complexities in kinematics and dynamics, complicating control and state estimation. This work presents a novel proprioceptive state estimator for tensegrity robots. The estimator initially uses the geometric constraints of 3-bar prism tensegrity structures, combined with IMU and motor encoder measurements, to reconstruct the robot’s shape and orientation. It then employs a contact-aided invariant extended Kalman filter with forward kinematics to estimate the global position and orientation of the tensegrity robot. The state estimator’s accuracy is assessed against ground truth data in both simulated environments and real-world tensegrity robot applications. It achieves an average drift percentage of 4.2%, comparable to the state estimation performance of traditional rigid robots. This state estimator advances the state-of-the-art in tensegrity robot state estimation and has the potential to run in real-time using onboard sensors, paving the way for full autonomy of tensegrity robots in unstructured environments.

Keywords: Modeling, Control, and Learning for Soft Robots, Flexible Robots, Motion Control, Remote Magnetic Actuation
Abstract: Magnetic continuum robots (MCRs) have become popular owing to their inherent advantages of easy miniaturization without requiring complicated transmission structures. The evolution of MCRs has significantly enhanced their dexterity. While much progress has been achieved, the quantitative index-based evaluation of deformability for different MPCs has not been addressed before. Here we use "deformability" to describe the capability for body deflection when an MCR forms different global shapes under an external magnetic field. Therefore, in this paper, we propose methodologies to design and control an MCR composed of modular axially magnetized segments. To guide robot MPC design, for the first time, we introduce a quantitative index-based evaluation strategy to analyze and optimize robot deformability. Additionally, a control framework with neural network-based controllers is developed to endow the robot with two control modes: the robot tip position and orientation (M1) and the global shape (M2). The excellent performance of the learnt controllers in terms of computation time and accuracy was validated via both simulation and experimental platforms.

Keywords: Modeling, Control, and Learning for Soft Robots, Reinforcement Learning, Machine Learning for Robot Control
Abstract: Accurate control of continuum robots is challenging, especially in the presence of external disturbances. To address this issue, reinforcement learning (RL) has been increasingly investigated in continuum robot control due to its online policy updating capability. However, the gap of Sim2Real transfer caused by inaccurate modeling results in RL implementation a dilemma. In this article, we propose a safety-critical Real2Sim2Real online RL framework, where the Real2Sim transfer is first achieved by the identification of a continuum robot using the Koopman operator. We further improve the training efficiency by introducing an imperfect demonstration into the RL framework. The offline policy is trained in simulations and then tested on a real continuum robot platform. During tests, the tracking performance is influenced by the hysteresis effect that cannot be captured by the Koopman operator. This results in a millimeter-level tracking root mean square error (RMSE). To address this issue, we online update the policy as well as the model, and the RMSE of the online controller outperforms the offline controller by 89.16% in free space and 85.70% under external payload, respectively.

Keywords: Soft Sensors and Actuators, Force and Tactile Sensing
Abstract: Stretchable sensors indicate promising prospects for soft robotics, medical devices, and human-machine interactions due to the high compliance of soft materials. Discrete sensing strategies, including sensor arrays and distributed sensors, are broadly involved in tactile sensors across versatile applications. However, it remains a challenge to achieve high spatial resolution with a self-decoupled capacity of magnitude with position from tactile stimuli and insensitivity to other off-axis stimuli for stretchable tactile sensors. Herein, we develop a stretchable tactile sensor with a parallel-assembled structure based on the proposed continuous spectral filtering principle, allowing superhigh spatial and force resolution for applied stimuli. This proposed sensor enables a high-linear spatial response (R^2＞0.996) even during stretching and bending, and high continuous spatial (7 μm) and force (5 mN) resolutions with design scalability and interaction robustness. We further demonstrate the sensors' performance by integrating them into a five-bar parallel mechanism for precise trajectory tracking (rotational resolution: 0.02°) in real time.

Keywords: Soft Sensors and Actuators, Perception-Action Coupling
Abstract: Shape memory alloy (SMA) is widely employed in developing actuators. However, the lack of sensing capabilities limits its application. This study presents a sensing-actuation integrated device based on SMA and triboelectric nanogenerator (TENG), achieving tactile sensing while maintaining the actuation performance. The proposed core-shell structure not only repurposes the SMA spring as a key component of actuation and sensing, but also effectively isolates the actuation current to prevent interference with the sensing signal. The aerogel-modified silicone composite layer is applied to the SMA to reduce temperature rise by 30.56%, ensuring the sensing performance. With a rapid response time of less than 31 ms and stable sensing performance exceeding 2000 cycles, the SMA-TENG actuator reliably detects dynamically varying forces and bending. Additionally, it generates a maximum actuation force of 3.21 N, which represents a 12.2% increase compared to a standard SMA spring, due to the pre-stress introduced by the composite layer. Moreover, it can actuate a displacement of 7.7 cm and exhibiting a power density of 7.15 × 10³ W/m³(at 0.84 V, 6 A). Finally, we validate its haptic sensing capability during actuation, demonstrating its potential towards interactive robotic systems.

Keywords: Soft Sensors and Actuators, Force and Tactile Sensing, Soft Robot Materials and Design
Abstract: Tactile sensing plays a crucial role to empower robotic hands with improved grasping and manipulating abilities. In this paper, we propose an anthropomorphic robotic hand design with dual air bag sensors integrated soft fingertips to achieve tactile sensing. The air bag sensor is low-cost, easy-to-build and deformable, and can be embedded in the fingertip, endows the hand with the ability to perceive and makes it have the mechanical complicance similar to the human fingertip. The air bag sensor exhibits high performance metrics, including a sensitivity of ~1.65 kPa/N, a minimum detection force of < 0.01 N, a response time of < 10 ms, and good stability and repeatability. The experimental results show that the proposed robotic hand performs well in surface texture detection, hard inclusion depth detection and object softness detection, as well as grasping tasks. By applying a machine learning algorithm to the experimental data, an accuracy of 0.767 and 0.898 was achieved in predicting hard inclusion depth and object hardness, respectively. This study provides a simple and effective tactile sensing solution for the design of anthropomorphic robotic hand, and may have possible applications such as end-effectors for humanoid robots or robotic palpation.

Keywords: Soft Sensors and Actuators, Robust/Adaptive Control, Flexible Robotics
Abstract: With the increasing demand for safe and efficient human-robot interaction in industrial applications, robotic manipulators with mixed rigid-elastic joints have gained significant attention, yet their control remains challenging due to inherent parameter uncertainties and complex dynamics. In this paper, an adaptive Cartesian position control for robotic manipulators with mixed rigid-elastic joints is presented. Adaptive controllers are designed to deal with uncertainties in the parameters of the motors, while robust control signals effectively cope with uncertainties of the links and stiffness of the elastic joints. Furthermore, a switching strategy between Cartesian space position control and joint space position control when the end-effector comes into the vicinity of the target point is proposed. This switching strategy helps to keep the pose of the robotic manipulator stable when the end-effector has approached the target point. Simulation results on a 6-DOF robotic manipulator demonstrate that the proposed control scheme can achieve the desired accuracy in position and maintain a stable pose when the end-effector reaches the target.

Keywords: Soft Sensors and Actuators, Gesture, Posture and Facial Expressions, Intention Recognition
Abstract: Hand motion monitoring plays a crucial role in fields such as human-machine interaction and rehabilitation training.Currently,electronic sensors are commonly used for hand motion monitoring.However,they are confronted with issues such as susceptibility to electromagnetic interference and sweat stains.Fiber Bragg Grating (FBG)sensors are small in size,highly sensitive,and possess good biocompatibility.In this paper,a flexible distributed Fiber Bragg Grating sensor is introduced.Emphasis is laid on the optimization and fabrication of the sensor,and performance tests are carried out on the fabricated sensor.To verify the potential of the sensor in hand motion monitoring,experiments are conducted.In the gesture recognition experiment,the Vision Transformer (ViT)model is utilized to classify eight types of gestures,and the final accuracy reaches 96.5%.In the wrist joint angle measurement experiment,the Pearson correlation coefficient between the physical angle and the measured angle is 0.985.In the grasping experiment,individual differences are reflected by the standard deviation during the grasping process.The experiments have demonstrated that the proposed sensor has the potential for monitoring hand motions.

Keywords: Soft Sensors and Actuators, Performance Evaluation and Benchmarking, Modeling, Control, and Learning for Soft Robots
Abstract: Soft robots rely on soft actuators, whose nonlinear responses are challenging to model, simulate, and integrate into designs. This complexity hinders the development of advanced soft robots and rigid mechanisms that use soft actuators for compliant actuation. To address this, in this study we present a framework for model development and actuator integration that offers both real and virtual environments for testing and validation. This framework includes high-fidelity digital twins and a physical integration bench. The digital twin allows for the testing of virtual actuator models in a validated digital environment, replicating various load profiles. The physical integration bench provides a safe, reproducible platform for validating both models and controllers under different loads, load profiles, and inertial conditions. Together, these tools enable rapid and reliable validation of actuator models and controllers, accelerating the development cycle of complex soft robots. We conclude by demonstrating the proposed workflow using liquid-gas phase transition actuators as a demonstrative test subject.

Keywords: Soft Sensors and Actuators, Actuation and Joint Mechanisms, Mechanism Design
Abstract: Osmosis-driven actuation offers a promising strategy for developing untethered, environmentally responsive soft and shape shifting mechanisms and robots. In this work, we explore the use of superabsorbent polymer (SAP) pellets as large-scale, shape-morphing actuators. Upon exposure to water, these approximately 2 mm diameter spherical pellets undergo a dramatic volumetric expansion, up to 300 times their initial volume, generating actuation forces of approximately 10 N under constrained conditions. We further demonstrate reversible cyclic actuation via controlled swelling-deswelling using ethanol-water solutions. Finally, we integrate these systems into a shape-morphing wheel design to enable adaptive locomotion that passively transitions between terrestrial and aquatic environments. Our findings demonstrate SAP-based osmotic actuators as an environmentally-driven solution for soft robotics, and shape-shifting soft hybrid mechanisms.

Keywords: Soft Sensors and Actuators, Force and Tactile Sensing
Abstract: While stretchable and compressible sensors are commonly used to enhance

the proprioception and exteroception of soft robots, their sensitivity and sensing range are constrained by their design and stiffness, often limiting their measurement capabilities. To overcome this limitation, this paper presents a novel approach that integrates stiffness modulation with piezoresistive sensing, specifically Electrical Impedance Tomography (EIT). This approach results in a new type of artificial skin, termed Variable Sensing Range Electrical Impedance Tomography Skin (VEITS), which features an adjustable sensing range. The VEITS comprises two layers: a top layer with variable stiffness achieved through granular jamming and a bottom EIT sensing layer. By applying a vacuum to the granular layer, the sensor stiffness can be increased, enabling it to measure higher contact forces. Specifically, applying a vacuum pressure of -80 kPa extends the contact force range by 37% compared to the unpressurized state. While increased stiffness temporarily reduce sensitivity to small loads, it does not affect the EIT localization accuracy. The VEITS simple structure makes it suitable

for large surface sensing on robots, and effectively expanding the sensing range. Though demonstrated with EIT and granular-jamming stiffness modulation, this novel principle is applicable to other strain-based sensors and stiffness modulation techniques.

Keywords: Surgical Robotics: Laparoscopy, Medical Robots and Systems, Computer Vision for Medical Robotics
Abstract: Minimally invasive surgery (MIS) offers significant benefits, such as reduced recovery time and minimised patient trauma, but poses challenges in visibility and access, making accurate 3D reconstruction a significant tool in surgical planning and navigation. This work introduces a robotic arm platform for efficient multi-view image acquisition and precise 3D reconstruction in MIS settings. We adapted a laparoscope to a robotic arm and captured ex-vivo images of several ovine organs across varying lighting conditions (operating room and laparoscopic) and trajectories (spherical and laparoscopic). We employed recently released learning-based feature matchers combined with COLMAP to produce our reconstructions. The reconstructions were evaluated against high-precision laser scans for quantitative evaluation. Our results show that whilst reconstructions suffer most under realistic MIS lighting and trajectory, two matching methods achieve close to sub-millimetre accuracy with 0.80 and 0.76mm Chamfer distances and 1.06 and 0.98mm RMSEs for ALIKED and GIM respectively. Our best reconstruction results occur with operating room lighting and spherical trajectories. Our robotic platform provides a tool for controlled, repeatable multi-view data acquisition for 3D generation in MIS environments, which can lead to new datasets necessary for novel learning-based surgical models.

Keywords: Surgical Robotics: Laparoscopy, Computer Vision for Medical Robotics, Computer Vision for Automation
Abstract: Autonomous soft tissue manipulation in robotic surgery remains challenging. Modeling the tool-tissue interaction, analyzing the tissue

structural deformation and monitoring the biomechanical status during surgery manipulation may benefit the development of autonomous surgery;

however, there are currently inadequate studies. We propose a vision-based surgery environment modeling framework to simultaneously reconstruct and track the forceps and the tissue, leveraging model-based pose estimation and scene flow-based mesh optimization. We also propose a digital twin based on the framework for continuously modeling the tool-tissue interaction and monitoring the deformation and

strain of the tissue surface. Quantitative and qualitative ex vivo experiments were conducted to evaluate the proposed system from various

perspectives. Results show that the deformation recovery accuracy is 0.38±0.30 mm with robustness to occlusion; the instrument pose estimation accuracy is 0.85±0.57 degree in rotation and 2.09±1.41 mm in

translation. The relative positioning between the tissue and the forceps can be correctly modeled in terms of contact detection. The system also correctly reveals the differences in strain distributions in two types of tool-tissue interaction, palpation and traction. With the proposed system, surgical robot systems with the perception of tissue deformation can be developed in the future for optimized and autonomous surgery.

Keywords: Surgical Robotics: Planning, Medical Robots and Systems
Abstract: Intraoperative optical navigation is widely utilized in robotic surgery systems. Typically, the observation pose of the optical tracking system (OTS) is manually adjusted and then fixed throughout the surgery. However, fixed OTS suffers from limited measurement volume (MV) and visual interferences, making consistent navigation challenging in clinics. In this paper, an operation-navigation dual-robot collaborative system is proposed for orthopedic surgeries. An extra navigation robot is introduced to actively adjust the observation pose of the OTS. A collaborative preoperative planning method is proposed for this dual-robot system, including osteotomy path planning of the operation robot and collaborative planning of the navigation robot. Firstly, osteotomy paths of the operation robot are generated according to the surgery regulations and the geometric features of the vertebral foramen. Secondly, based on the generated osteotomy paths, the collaborative planning of the navigation robot is formulated into a multi-objective optimization problem to find the optimal poses of the OTS for each osteotomy plane. Compared with fixed OTS, active navigation is capable of keeping all the targets within the MV of the OTS throughout the surgery. Semi-laminectomy on a human spine phantom is adopted as an example to experimentally evaluate the effectiveness of the proposed method.

Keywords: Surgical Robotics: Planning, Task Planning, Surgical Robotics: Laparoscopy
Abstract: The rise of Large Language Models (LLMs) has impacted research in robotics and automation. While progress has been made in integrating LLMs into general robotics tasks, a noticeable void persists in their adoption in more specific domains such as surgery, where critical factors such as reasoning, explainability, and safety are paramount. Achieving autonomy in robotic surgery, which entails the ability to reason and adapt to changes in the environment, remains a significant challenge. In this work, we propose a multi-modal LLM integration in robot-assisted surgery for autonomous blood suction. The reasoning and prioritization are delegated to the higher-level task-planning LLM, and

the motion planning and execution are handled by the lower-level deep reinforcement learning model, creating a distributed agency between the

two components. As surgical operations are highly dynamic and may encounter unforeseen circumstances, blood clots and active bleeding were introduced to influence decision-making. Results showed that using

a multi-modal LLM as a higher-level reasoning unit can account for these surgical complexities to achieve a level of reasoning previously unattainable in robot-assisted surgeries. These findings demonstrate the potential of multi-modal LLMs to significantly enhance contextual understanding and decision-making in robotic-assisted surgeries, marking a step toward autonomous surgical systems.

Keywords: Surgical Robotics: Steerable Catheters/Needles, Tendon/Wire Mechanism
Abstract: Robot-assisted gastrointestinal endoscopic surgery (GES) requires flexible manipulators to possess a compact dimension and stiffness tuning capability. Current stiffness-tunable miniature manipulators (STMM) using tendon-sheath mechanism (TSM) experience the problem of stiffness-influenced hysteresis. To address this, we propose the first stiffness-dependent Modified Generalized Prandtl-Ishlinski (MGPI) model and a specific compensation strategy. Firstly, we analyzed the stiffness tuning mechanism and extracted the stiffness parameter. Based on this, the analytical hysteresis model and its inverse function were built and verified in simulations and experiments, which increases the fitting accuracy of the hysteresis. Then, with the assistance of the compensation strategy, the real-time input-output relationship of the STMM's stiffness-tunable joints can be approximately linear. The average errors of trajectory tracking achieve a significant improvement of over 85%. This work provides a new method to model and compensate for the dynamic hysteresis in flexible endoscopic robots with the stiffness-tunable TSM.

Keywords: Surgical Robotics: Steerable Catheters/Needles, Surgical Robotics: Planning, Motion and Path Planning, Nonholonomic Motion Planning
Abstract: Steerable needles are minimally invasive devices that enable novel medical procedures by following curved paths to avoid critical anatomical obstacles. We introduce a new start pose robustness metric for steerable needle motion plans. A steerable needle deployment typically consists of a physician manually placing a steerable needle at a precomputed start pose on the surface of tissue and handing off control to a robot, which then autonomously steers the needle through the tissue to the target. The handoff between humans and robots is critical for procedure success, as even small deviations from a planned start pose change the steerable needle's workspace. Our metric is based on a novel geometric method to efficiently compute how far the physician can deviate from the planned start pose in both position and orientation such that the steerable needle can still reach the target. We evaluate our metric through simulation in liver and lung scenarios. Our evaluation shows that our metric can be applied to plans computed by different steerable needle motion planners and that it can be used to efficiently select plans with large safe start regions.

Keywords: Robot Safety, Surgical Robotics: Planning, Surgical Robotics: Laparoscopy
Abstract: Resection of pathological tissue is a common procedure in surgical oncology for treating tumors. In robot-assisted electrosurgery, the use of predefined markers to guide autonomous robotic resection is gaining traction. Accurate tracking of these markers and minimizing electrocautery damage are critical for the safe and effective autonomous resection of tumors. This paper introduces a safety enhanced autonomous resection method for laparoscopic surgery, designed to mitigate the risks posed by tissue deformation during the resection process. Initially, we pre-plan the cutting path and design a switching strategy for navigation waypoints based on a preview tracking mechanism. Then, we develop a depth-fused navigation controller and a safe withdrawal motion controller. Next, an inertial tracking mechanism is established to evaluate tissue deformation over short periods. Finally, we develop a confidence generator to fuse the two controllers, ensuring that tissue deformation during the resection process does not cause additional electrocautery damage. Simulation and phantom experiments were conducted, demonstrating the effectiveness of our proposed method. This work represents a significant step toward achieving autonomous robotic resection.

Keywords: Dual Arm Manipulation, Kinematics, Optimization and Optimal Control
Abstract: This paper presents ETA-IK, a novel Execution-Time-Aware Inverse Kinematics method tailored for dual-arm robotic systems. The primary goal is to optimize motion execution time by leveraging the redundancy of the entire system, specifically in tasks where only the relative pose of the robots is constrained, such as dual-arm scanning of unknown objects. Unlike traditional IK methods using surrogate metrics, our approach directly optimizes execution time while implicitly considering collisions. A neural network based execution time approximator is employed to predict time-efficient joint configurations while accounting for potential collisions. Through experimental evaluation on a system composed of a UR5 and a KUKA iiwa robot, we demonstrate significant reductions in execution time. The proposed method outperforms conventional approaches, showing improved motion efficiency without sacrificing positioning accuracy.

Keywords: Kinematics, Machine Learning for Robot Control, Redundant Robots
Abstract: Data-Driven Inverse Kinematics (DDIK) solvers emerged as promising Inverse Kinematics (IK) methods for reliably approximating the IK of robotic manipulators. However, these solvers remain heavily robot-dependent, where for each robot of interest, a network needs to be trained in an one-solver-one-robot framework. In this paper, we build on our previous work on Learning-By-Example (LBE) for DDIK, and introduce an one-solver-many-robots framework; where a single neural network is used to predict the IK of multiple robots -- mainly with 6 and 7 Degrees of Freedom (DoF). In our LBE approach, the neural network input includes an example of joint-pose tuple (e.g. any previous joint and corresponding pose tuple in the path) along with the queried pose as the same network outputs the desired robot joint configuration. Here, we investigate five network architectures: a Plain Multilayer Perceptron (MLP), a Residual-based MLP (RMLP), a Densely Connected MLP (DMLP), and two transformers inspired by Generative Pre-trained Transformer (GPT) and tested them using 3 diverse datasets with 20 real-world robotic arms with 6 and 7 DoF. Our experimental results demonstrate that a single lightweight, LBE-based DDIK solver can reliably predict the IK for multiple and hitherto unseen robots, within each of the 6 or 7DoF family as well as across both 6 and 7DoF robot families with position errors below 1mm and orientation errors below 1deg. Additionally, we compare all proposed LBE-based DDIK solvers with three established numerical IK solvers: Selectively Damped Least-Squares (SD), Singular Value Filtering (SVF), and Mixed Inverse (MX) and observe that our LBE-based DDIK solvers achieve comparable accuracy, with the advantage of being a one-solver-many-robots framework.

Keywords: Kinematics, Probability and Statistical Methods, Localization
Abstract: This article presents a nonlinear estimator on matrix Lie group that performs a fast unscented transformation with natural evolution of sigma points from a geometric perspective. Different from the existing methods, the proposed method preserves the original dynamic equations on the manifold, which greatly reduces the computational time without changing the system configuration space or reducing the number of sigma points. We provide a new state propagation and update method of UKF on manifolds, where only the mean state is involved, and the remaining sigma points are calculated and propagated as incremental information based on the state of the previous step, according to the fundamental property of geometric filtering on the Lie group. Moreover, by decoupling the parameter variables, we investigate the upper limit of the efficiency improvement of the proposed algorithm compared to the traditional unscented transformation in different situations. Finally, two representative experiments are conducted to validate the proposed theory, the experiments show that the proposed method achieves desirable performance with much higher computational efficiency as compared with the existing UKF algorithms on manifolds.

Keywords: Kinematics, Additive Manufacturing, Industrial Robots
Abstract: In this paper, we present a concurrent and scalable trajectory optimization method to improve the quality of robot-assisted manufacturing. Our method simultaneously optimizes tool orientations, kinematic redundancy, and waypoint timing on input toolpaths with large numbers of waypoints to improve kinematic smoothness while incorporating manufacturing constraints. Differently, existing methods always determine them in a decoupled manner. To deal with the large number of waypoints on a toolpath, we propose a decomposition-based numerical scheme to optimize the trajectory in an out-of-core manner, which can also run in parallel to improve the efficiency. Simulations and physical experiments have been conducted to demonstrate the performance of our method in examples of robot-assisted additive manufacturing.

Keywords: Kinematics, Robust/Adaptive Control, Redundant Robots
Abstract: Kinematic control is one of the fundamental issues in the study of redundant robotic manipulators, and various schemes that utilize the pseudoinverse of the Jacobian have been crafted in this field. However, these schemes may be ineffective due to the lack of suppressing additive noise, especially the harmonic noise widely encountered in practice. This article proposes a new kinematic control scheme for redundant robotic manipulators affected by harmonic noise, aiming to overcome the identified limitations. Such a scheme, which incorporates the adaptive learning mechanism based on harmonic frequency, can simulate the harmonic noise, suppress its disturbance, and eventually realize the effective control purpose. It is then theoretically proven that the Cartesian error generated by the proposed scheme exhibits the convergence property, thus guaranteeing the control performance on redundant robotic manipulators even in the presence of harmonic noise. Simulation and experiment results under PA10 and Panda robotic manipulators further verify the efficacy and superiority of the proposed kinematic control scheme.

Keywords: Integrated Planning and Control, Collision Avoidance, Constrained Motion Planning
Abstract: Safe autonomous navigation in unknown environments remains a critical challenge for robots with limited sensing capabilities. While safety-critical control techniques, such as Control Barrier Functions (CBFs), have been proposed to ensure safety, their effectiveness relies on the assumption that the robot has complete knowledge of its surroundings. In reality, robots often operate with restricted field-of-view and finite sensing range, which can lead to collisions with unknown obstacles if the planner is agnostic to these limitations. To address this issue, we introduce the Visibility-Aware RRT* algorithm that combines sampling-based planning with CBFs to generate safe and efficient global reference paths in partially unknown environments. The algorithm incorporates a collision avoidance CBF and a novel visibility CBF, which guarantees that the robot remains within locally collision-free regions, enabling timely detection and avoidance of unknown obstacles. We conduct extensive experiments interfacing the path planners with two different safety-critical controllers, wherein our method outperforms all other compared baselines across both safety and efficiency aspects.

Keywords: Planning, Scheduling and Coordination, Sustainable Production and Service Automation
Abstract: 经济全球化时代，分布式 异构焊接车间调度问题（DHWSP）有 被考虑过。同时，在一些实际生产中 场景，部分作业只能在某些情况下处理 指定工厂（即工厂资格）和 不可避免的不可控系统干扰（例如 机器维护和人为因素）导致不确定 作业的处理时间。然而，研究考虑 DHWSP与工厂的处理时间不确定 资格仍未探索。考虑优势 区间类型 2 模糊数 （IT2FN） 表示 高度不确定性，IT2FN 的概念是 引入以解决不确定的处理时间。然后，在 绿色制造的背景，本文对 一种高能效的分布式异构2型模糊 焊接车间调度问题与工厂资格 （EDHFWSP-FE）。基于学习的混合人工蜂群 算法 （LHABC） 旨在最小化总能量 EDHFWSP-FE 中的消耗和制造跨度。在 LHABC 中，一个 协同初始化呈现，打造优秀 初步解决方案。在使用蜜蜂阶段，Q 学习 基于方法的开发是为了帮助解决方案选择 优越的邻里结构。在旁观蜜蜂阶段，一个 提出可变邻域搜索（VNS）来挖掘 有前途的社区解决方案。在侦察蜂阶段，一个 基于分布算法（EDA）的估

Keywords: Deep Learning in Grasping and Manipulation, Data Sets for Robot Learning, Grasping
Abstract: Robotic grasping guided by natural language instructions faces challenges due to ambiguities in object descriptions and the need to interpret complex spatial context. Existing visual grounding methods often rely on datasets that fail to capture these complexities, particularly when object categories are vague or undefined. To address these challenges, we make three key contributions. First, we present an automated dataset generation engine for visual grounding in tabletop grasping, combining procedural scene synthesis with template-based referring expression generation, requiring no manual labeling. Second, we introduce the RefGrasp dataset, featuring diverse indoor environments and linguistically challenging expressions for robotic grasping tasks. Third, we propose a visually grounded dexterous grasping framework with continuous grasp generation, validated through extensive real-world robotic experiments. Our work offers a novel approach for language-guided robotic manipulation, providing both a challenging dataset and an effective grasping framework for real-world applications.

Keywords: Reinforcement Learning, Learning from Demonstration, Deep Learning in Grasping and Manipulation
Abstract: Prior knowledge vastly exists in the automation industry, especially for tasks like pick-and-place, where simple programmatic demonstrations with online generation ability can be acquired easily. How to learn a policy faster with higher flexibility and generalization ability based on these demonstrations is a question to be answered. End-to-end target learning and imitation learning are widely discussed in previous works. Here, we focus on the online generation ability of the demonstration and propose a demo injection method based on actor-critic off-policy reinforcement learning (RL) for the interaction and policy optimization phase. We conduct experiments and an ablation study based on four research questions around a two-finger reactive grasping task with a Panda robot. The result shows our proposed injection method increases the training stability, strongly reduces the time to convergence and benefits sim-2-real transfer with smooth motion.

Keywords: AI-Enabled Robotics, Manipulation Planning, Deep Learning Methods
Abstract: The proficiency of robots in cloth manipulation is crucial for their potential widespread deployment in household service contexts, with the task of unfolding cloth being particularly indispensable. Unlike rigid objects, cloth has a high-dimensional state space, which poses significant challenges for robotic operations. This paper presents a robotic framework that integrates dynamic and static operations for cloth unfolding. Dynamic operations are introduced in a single-arm scenario, employing gravity to expedite flattening. Initially, we define the classification of cloth states and operational skills. Subsequently, in skill selection, a Large Language Model (LLM) is utilized to make decisions based on the current state, selecting skills appropriate for the given situation. For the determination of operation points, a cloth region segmentation network extracts key features of the cloth, and the final operation points are determined through geometric analysis of the masks. Experiments on a real robot demonstrate that our method can successfully unfold cloths of various initial conditions, colors, sizes, textures, shapes and materials, achieving over 95% coverage - defined as the ratio of the current area of the fabric to its fully expanded area - thereby proving the effectiveness of the combined dynamic and static operation strategy. Furthermore, this method significantly enhances the efficiency of cloth unfolding, completing the task within ten actions, whereas other methods require dozens of operations, greatly reducing the required operational complexity.

Keywords: Deep Learning in Grasping and Manipulation
Abstract: In complex scenarios where typical pick-and-place techniques are insufficient, often non-prehensile manipulation can ensure that a robot is able to fulfill its task. However, non-prehensile manipulation is challenging due to its underactuated nature with hybrid-dynamics, where a robot needs to reason about an object's long-term behavior and contact-switching, while being robust to contact uncertainty. The presence of clutter in the workspace further complicates this task, introducing the need to include more advanced spatial analysis to avoid unwanted collisions. Building upon prior work on reinforcement learning with multimodal categorical exploration for planar pushing, we propose to incorporate location-based attention to enable robust manipulation in cluttered scenes. Unlike previous approaches addressing this obstacle avoiding pushing task, our framework requires no predefined global paths and considers the desired target orientation of the manipulated object. Experimental results in simulation as well as with a real KUKA iiwa robot arm demonstrate that our learned policy manipulates objects successfully while avoiding collisions through complex obstacle configurations, including dynamic obstacles, to reach the desired target pose.

Keywords: Deep Learning in Grasping and Manipulation, Perception for Grasping and Manipulation, Multifingered Hands
Abstract: In our daily life, we often encounter objects that are fragile and can be damaged by excessive grasping force, such as fruits. For these objects, it is paramount to grasp gently—not using the maximum amount of force possible, but rather the minimum amount of force necessary. This paper proposes using visual, tactile, and auditory signals to learn to grasp and regrasp objects stably and gently. Specifically, we use audio signals as an indicator of gentleness during the grasping, and then train an end-to-end action-conditional model from raw visuo-tactile inputs that predicts both the stability and the gentleness of future grasping candidates, thus allowing the selection and execution of the most promising action. Experimental results on a multi-fingered hand over 1,500 grasping trials demonstrated that our model is useful for gentle grasping by validating the predictive performance (3.27% higher accuracy than the vision-only variant) and providing interpretations of their behavior. Finally, real-world experiments confirmed that the grasping performance with the trained multi-modal model outperformed other baselines (17% higher rate for stable and gentle grasps than vision-only). Our approach requires neither tactile sensor calibration nor analytical force modeling, drastically reducing the engineering effort to grasp fragile objects. Dataset and videos are available at https://lasr.org/research/gentle-grasping.

Keywords: Deep Learning in Grasping and Manipulation, Perception for Grasping and Manipulation, Service Robotics
Abstract: Physical manipulation of garments is often crucial when performing fabric-related tasks, such as hanging garments. However, due to the deformable nature of fabrics, these operations remain a significant challenge for robots in household, healthcare, and industrial environments. In this paper, we propose GraphGarment, a novel approach that models garment dynamics based on robot control inputs and applies the learned dynamics model to facilitate garment manipulation tasks such as hanging. Specifically, we use graphs to represent the interactions between the robot end-effector and the garment. GraphGarment uses a graph neural network (GNN) to learn a dynamics model that can predict the next garment state given the current state and input action in simulation. To address the substantial sim-to-real gap, we propose a residual model that compensates for garment state prediction errors, thereby improving real-world performance. The garment dynamics model is then applied to a model-based action sampling strategy, where it is utilized to manipulate the garment to a reference pre-hanging configuration for garment-hanging tasks. We conducted four experiments using six types of garments to validate our approach in both simulation and real-world settings. In simulation experiments, GraphGarment achieves better garment state prediction performance, with a prediction error 0.46 cm lower than the best baseline. Our approach also demonstrates improved performance in the garment-hanging simulation experiment—with enhancements of 12%, 24%, and 10%, respectively. Moreover, real-world robot experiments confirm the robustness of sim-to- real transfer, with an error increase of 0.17 cm compared to simulation results. Supplementary material is available at: https://sites.google.com/view/graphgarment.

Keywords: Deep Learning in Grasping and Manipulation, AI-Based Methods, Deep Learning Methods
Abstract: Visual augmentation has become a crucial technique for enhancing the visual robustness of imitation learning. However, existing methods are often limited by prerequisites such as camera calibration or the need for controlled environments (e.g., green screen setups). In this work, we introduce RoboEngine, the first plug-and-play visual robot data augmentation toolkit. For the first time, users can effortlessly generate physics- and task-aware robot scenes with just a few lines of code. To achieve this, we present a novel robot scene segmentation dataset, a generalizable high-quality robot segmentation model, and a fine-tuned background generation model, which together form the core components of the out-of-the-box toolkit. Using RoboEngine, we demonstrate the ability to generalize robot manipulation tasks across six entirely new scenes, based solely on demonstrations collected from a single scene, achieving a more than 200% performance improvement compared to the no-augmentation baseline. All datasets, model weights, and the toolkit are released https://roboengine.github.io/.

Keywords: Reinforcement Learning, Human Factors and Human-in-the-Loop, Manipulation Planning
Abstract: A novel efficient reinforcement learning paradigm combining human knowledge, model-based and model-free methods is presented for optimal robot manipulation control during complex multi-phase robot manipulation tasks, e.g., the pegin-hole tasks with tight fit and nut-and-bolt assembly. Firstly, human demonstration is conducted to collect the data during successful robot manipulation, and manipulation phase estimation method integrating with human knowledge is presented to obtain the higher-level planning of the multi-phase robot manipulation tasks. Typical robot manipulation tasks can usually be decomposed into three types of phases, namely free motion, discontinuous contact, and continuous contact. For phase with free motion, the motion planning method is utilized for generating smooth trajectory. For phase with discontinuous contact in the axes of interest during the pre-manipulation process, the rule-based model-free method, namely the Policy Gradients with Human-Guided Parameter-based Exploration (PGHGPE) method is utilized. For the manipulation phase with continuous contacts, the model-based method is utilized because of its higher sample efficiency. Finally, the simulation and experimental studies verify the effectiveness of the presented algorithm.

Keywords: Motion and Path Planning, Path Planning for Multiple Mobile Robots or Agents, Cooperating Robots
Abstract: We propose a continuous gathering scheme based on the swept volume to address the challenges involved in planning a tethered robot duo to efficiently collect marine debris. Specifically, we model the tethered robot duo by constructing a double-layer U-shape, and then apply an object-aware optimization approach that leverages the swept volume signed distance field (SVSDF) to guide trajectory optimization, promoting complete object collection while maintaining a continuous and collision-free gathering motion. Existing algorithms either fail to fully address key challenges, such as assuming an unrealistically infinite tether length or incurring high computational costs. In contrast, our proposed method, by adopting the double-layer U-shape technique, effectively manages tether length constraints and preserves the tether shape, ensuring feasible collection. By utilizing the SVSDF technique to guide the trajectory optimization process, we maximize the swept coverage of objects while minimizing that of obstacles. This enables complete object coverage, avoids collisions, and prevents the tether from becoming trapped by obstacles during the collection process. Moreover, we propose a set of metrics for this gathering planning problem and validate the generated trajectories in simulation, using a collision-free multi-UAV information-gathering approach to efficiently estimate the target area. Simulations demonstrate that our proposed method achieves superior, resolution-independent gathering performance compared to existing algorithms.

Keywords: Motion and Path Planning, Reactive and Sensor-Based Planning, Force Control, Ergodic Control
Abstract: In this article, we present a feedback control method for tactile coverage tasks, such as cleaning or surface inspection. These tasks are challenging to plan due to complex, continuous physical interactions. In these tasks, the coverage target and progress can be easily measured using a camera and encoded in a point cloud. We propose an ergodic coverage method that operates directly on point clouds, guiding the robot to spend more time on regions requiring more coverage. For robot control and contact behavior, we use geometric algebra to formulate a task-space impedance controller that tracks a line while simultaneously exerting a desired force along that line. We evaluate the performance of our method in kinematic simulations and demonstrate its applicability in real-world experiments on kitchenware. Our source codes, experimental data, and videos are available as open access at url{https://sites.google.com/view/tactile-ergodic-control/}.

Keywords: Motion and Path Planning, Reactive and Sensor-Based Planning, Collision Avoidance
Abstract: This work presents the implementation and evaluation of a real-time collision-avoiding motion planning algorithm for highly dynamic environments. By combining short-horizon obstacle estimation with the robot constraints, our method implements collision-avoiding situational-aware motion planning by heuristically exploring multiple relevant paths. Directly feeding the current motion setpoint of the path into the low-level controller closes the loop between motion planning and low-level control, ensuring constraint-aware execution. Its practical implementation in physical robots in dynamic RoboCup-like scenarios validated its effectiveness, with low computational costs enabling fast adaptation to changing environments. Furthermore, the capabilities of our motion planner were demonstrated during RoboCup 2024 and practice matches.

Keywords: Motion and Path Planning, Path Planning for Multiple Mobile Robots or Agents, Probability and Statistical Methods
Abstract: Robot navigation in dynamic crowded environments poses a significant challenge due to the uncertainties inherent in the obstacles' model. In this work, we propose a risk-adaptive approach based on the CVaR Barrier Function (CVaR-BF): 1) We adjust the risk level to adopt the minimally-necessary risk when navigating through crowds, thereby achieving both safety and feasibility under uncertainties. 2) We characterize collision probability by evaluating the relative state between the robot and the obstacle and incorporate this risk assessment into a novel Control Barrier Function (CBF) design, termed the dynamic zone-based CBF. Paired with the risk-level adaptation, this dynamic zone-based CBF yields an enhanced responsiveness to obstacles in highly dynamic scenarios. Comparisons and ablation studies demonstrate that our method outperforms existing social navigation approaches and validate the effectiveness of the individual components of our adaptive strategy.

Keywords: Motion and Path Planning, Reactive and Sensor-Based Planning, Surveillance Robotic Systems
Abstract: The utilization of Unmanned Ground Vehicles (UGVs) for patrolling industrial sites has expanded significantly. These UGVs typically are equipped with perception systems, e.g., computer vision, with limited range due to sensor limitations or site topology. High-level control of the UGVs requires Coverage Path Planning (CPP) algorithms that navigate all relevant waypoints and promptly start the next cycle. In this paper, we propose the novel Fast-Revisit Coverage Path Planning (FaRe-CPP) algorithm using a greedy heuristic approach to propose waypoints for maximum coverage area and a random search-based path optimization technique to obtain a path along the proposed waypoints with minimum revisit time. We evaluated the algorithm in a simulated environment using Gazebo and a camera-equipped TurtleBot3 against a number of existing algorithms. Compared to their average path lengths and revisit times, our FaRe-CPP algorithm showed a reduction of at least 21% and 33%, respectively, in these highly relevant performance indicators.

Keywords: Motion and Path Planning, Reactive and Sensor-Based Planning
Abstract: Autonomous exploration in unknown environments requires estimating the information gain of an action to guide planning decisions. While prior approaches often compute information gain at discrete waypoints, pathwise integration offers a more comprehensive estimation but is often computationally challenging or infeasible and prone to overestimation. In this work, we propose the Pathwise Information Gain with Map Prediction for Exploration (PIPE) planner, which integrates cumulative sensor coverage along planned trajectories while leveraging map prediction to mitigate overestimation. To enable efficient pathwise coverage computation, we introduce a method to efficiently calculate the expected observation mask along the planned path, significantly reducing computational overhead. We validate PIPE on real-world floorplan datasets, demonstrating its superior performance over state-of-the-art baselines. Our results highlight the benefits of integrating predictive mapping with pathwise information gain for efficient and informed exploration.

Keywords: Motion and Path Planning, Reinforcement Learning
Abstract: Autonomous navigation of unmanned ground vehicles (UGVs) in dynamic scenes is a challenging task that requires them to avoid obstacles and move toward the goal simultaneously. This paper proposes ALVO, an adaptive learning policy that leverages velocity obstacles for UGV navigation. ALVO employs an adaptive gating-based mechanism for reactive obstacle avoidance, which enables the UGV to either slow down or proactively navigate around obstacles based on the relative importance of the environmental state and the goal. A reward function based on velocity obstacles is also designed to guide the UGV to navigate toward the goal while avoiding obstacles. Extensive experiments demonstrate that ALVO outperforms the competing approaches in various dynamic environments. We also implemented our method on a real UGV and showed that it performed well in real-world scenarios.

Keywords: Motion and Path Planning, Soft Robot Applications, Biologically-Inspired Robots
Abstract: Many path planning algorithms are proposed and employed for cable-driven manipulators (CDMs). However, most of them only consider single-target-point tasks. For multi-target-point tasks, CDMs need to repeat the planning and following of single point tasks. This is feasible but not optimal in terms of the distance and time needed by CDMs to complete such tasks. To solve this problem, this work designs a novel two-stage multi-target-point path planning (MPP) method. In the first stage, an improved rapidly exploring random tree (RRT)-A* algorithm that considers CDMs' features is used to preplan passable paths between each target and a start point. In the second one, in order to avoid CDM's repetitively moving along similar preplanned paths, a cosine similarity theory is used, for the first time, to integrate these paths. Furthermore, an indicator named path cost is defined to evaluate paths. This indicator takes into account CDMs' constraints, paths' lengths, and energy consumption. Simulations are conducted to compare MPP with some classical and recently developed algorithms. The results shows that it well outperform them in terms of path length and tracking time. Furthermore, the proposed method is verified by experiments in a 17 degree-of-freedom CDM prototype.

Keywords: Computer Vision for Medical Robotics, Medical Robots and Systems
Abstract: In computer-assisted orthopedic surgery (CAOS), accurate point set registration is essential for enhancing surgical accuracy. However, the sparse and low-overlap nature of intraoperative point sets presents significant challenges for reliable registration. To deal with these challenges, we propose a novel registration-after-completion framework, where the intraoperative point set is first completed, after which the two full point sets are registered. Our main contributions include the follows. First, we propose a progressive two-stage strategy to progressively complete the sparse and partial intraoperative point set. Second, considering that 1) intra-operative point set contains noise 2) the point completion process is not perfect, and 3) the resolution of preoperative image is limited, we adopt the bidirectional hybrid mixture models (HMMs) to represent the point set pairs and formulate the probabilistic registration network. In the proposed novel correspondence network where a dual-path cross-attention mechanism is adopted for feature fusion and a clustering mechanism is leveraged for calculating point-to-mixture correspondences. Furthermore, the bidirectional registration mechanism is leveraged to compute the transformation based on estimated correspondences. Third, we have extensively validated the proposed approach on various datasets and bone phantoms. Our experiments on 1399 human femur and 1301 hip models demonstrate that our method achieves state-of-the-art performance across overlap rates from 15% to 35% and at various point counts (i.e., 25, 50, and 100 points) under conditions with less than 50% overlap. Additionally, real phantom experiments on femur and hip models validate the method’s performance in simulated surgical scenarios. Experiments on ModelNet40 further confirmed our method's effectiveness and generalizability.

Keywords: Computer Vision for Medical Robotics, Computer Vision for Automation
Abstract: Robot-assisted endovascular navigation provides significant advantages, including reduced radiation exposure for surgeons and improved patient safety. However, a major challenge is to control curvilinear instruments like guidewires precisely for smooth and accurate navigation while adapting to anatomical variations and external forces. Traditional segmentation-based approaches struggle with real-time prediction of the guidewire’s evolving shape, limiting their effectiveness in navigation tasks. In this paper, we propose SplineFormer, an explainable transformer network that predicts the continuous, structured representation of the guidewire as a B-spline. This formulation enables a compact, smooth, and explainable state representation that facilitates downstream navigation. By leveraging SplineFormer’s predictions within an imitation learning framework, our system successfully performs autonomous endovascular navigation. Experimental results show that SplineFormer achieves a 50% success rate when fully autonomously cannulating the Brachiocephalic Artery in a real robotic setup, demonstrating its potential for improved autonomous navigation in endovascular interventions.

Keywords: Computer Vision for Medical Robotics, Surgical Robotics: Planning, Medical Robots and Systems
Abstract: Invasive flexible neural electrodes are becoming increasingly prevalent in monitoring and modulating brain neural activity, necessitating the precise and minimally invasive implantation of these electrodes to a depth of a few millimeters beneath the cerebral surface. Although Neuralink has pioneered robot-assisted neural electrode implantation guided by microscopy, it currently lacks the ability to detect non-cerebral surface microvessels that are invisible under the white-light microscope, leading to inaccurate implantation planning and a high risk of trauma. To address this limitation, we introduce a vascular-enhanced strategy that fuses intraoperative white-light microscopy and preoperative photoacoustic microscopy and applies the fusion results to our established microsurgical robotic system for brain electrode implantation. Specifically, a multi-modality data preprocessing pipeline is devised to extract representative features, and a 2.5D fusion network that incorporates a depth encoding mechanism is proposed to predict cross-modality correspondence. The enhanced fusion results are utilized for implantation planning and intraoperative guidance during in vivo surgical procedures. Both quantitative and qualitative results are presented to demonstrate the effectiveness of our proposed cross-modality fusion methods. Furthermore, in vivo surgical implementations on mice underscore the potential of the proposed approach for achieving more precise and minimally invasive brain electrode implantation.

Keywords: Computer Vision for Medical Robotics, Object Detection, Segmentation and Categorization, Deep Learning Methods
Abstract: The semi-supervised medical image segmentation with a few annotated data can provide significant help in robot-assisted surgery. This step plays a pivotal role in identification of pathological regions, more appropriate planning of surgical procedures, and so on. In this work, we develop an evidence-based tri-branch cross-pseudo supervision model, which integrates evidence-based uncertainty estimation and multi-branch cross supervision to bolster the effectiveness of semi-supervised learning. The overall framework consists of a vanilla network and an evidential dual-branch network. Two evidential branches EPB and ERB are proposed to complement each other and improve the quality of pseudo-labels. The EPB places more focus on classification accuracy at the pixel level and the ERB emphasizes the similarity and overall integrity of the segmented regions. Then, a novel cross-pseudo supervision strategy among the three branches is designed, to guarantee that valuable and diverse unlabeled knowledge is explored and transferred for segmentation improvement. The effectiveness of the proposed method was verified on the ACDC dataset, achieving outstanding performance compared with other state-of-the-art methods. In addition, we conducted ablation study to validate the effectiveness of the evidential branches (EPB and ERB) and tri-branch cross-supervision strategy, respectively.

Keywords: Computer Vision for Medical Robotics, Deep Learning for Visual Perception, Vision-Based Navigation
Abstract: Monocular depth estimation and ego-motion estimation are significant tasks for scene perception and navigation in stable, accurate and efficient robot-assisted endoscopy. To tackle lighting variations and sparse textures in endoscopic scenes, multiple techniques including optical flow, appearance flow and intrinsic image decomposition have been introduced into the existing methods. However, the effective training strategy for multiple modules are still critical to deal with both of illumination reflectance and information interference for self-supervised depth estimation in endoscopy. Therefore, a novel framework with multistep efficient finetuning is proposed in this work. In each epoch of end-to-end training, the process is divided into three steps, including optical flow registration, multiscale image decomposition and multiple transformation alignments. At each step, only the related networks are trained without interference of irrelevant information. Based on parameter-efficient finetuning on the foundation model, the proposed method achieves state-of-the-art performance on self-supervised depth estimation on SCARED dataset and zero-shot depth estimation on Hamlyn dataset, with 4%-10% lower error. The code of this work has been pre-released on https://github.com/BaymaxShao/EndoMUST.

Keywords: Computer Vision for Medical Robotics, Medical Robots and Systems
Abstract: 随着手术机器人在临床实践中的使用越来越多，增强其处理多模态医学图像的能力已成为一项关键的研究挑战。尽管传统的医学图像融合方法在提高融合精度方面取得了进展，但在实时性、细粒度特征提取和边缘保留方面仍面临重大挑战。在本文中，我们介绍了TTTFusion，这是一种基于测试时间训练（TTT）的图像融合策略，可在推理过程中动态调整模型参数，以有效地融合多模态医学图像。通过在测试阶段调整模型，我们的方法根据输入的图像数据优化参数，从而提高融合结果的准确性和更好的细节保留。实验结果表明，与传统的融合方法相比，TTTFusion显著提高了多模态图像的融合质量，特别是在细粒度特征提取和边缘保留方面。这种方法不仅提高了图像融合的准确性，还为手术机器人的实时图像处理提供了一种新颖的技术解决方案。

Keywords: Computer Vision for Medical Robotics, Medical Robots and Systems, AI-Based Methods
Abstract: With the continuous advancement of minimally invasive surgery, the incisions have become increasingly smaller, leading to a proportional convergence between the physical dimensions of surgical instruments and the operational space. This phenomenon has exacerbated the issue of visual obstruction caused by the overlapping of the instruments, which now stands as a significant technical impediment to the enhancement of precision in minimally invasive procedures. Vanilla approaches rely on deep learning-based image inpainting techniques to address this issue, but their results are unreliable for surgeons making decisions. This work rethinks hardware design, sacrificing resolution to increase the view of cameras, effectively filling the instrument occluded areas through multi-view technology. Subsequently, a super-resolution method is employed to restore the details of the inpainted images. This innovative approach transforms the uncertainty of deep learning from a large-range pixel inference problem into a more controllable pixel interpolation task, thereby significantly enhancing the reliability and accuracy of the repaired results. Taking spinal endoscopic surgery as an application scenario, we designed a miniature multi-view endoscope system tailored to the specific needs of this surgery. Phantom experiments are conducted to verify the feasibility of the proposed instrument transparency method. The results demonstrate the potential of the proposed method for improving minimally invasive surgery.

Keywords: Prosthetics and Exoskeletons, Wearable Robotics, Medical Robots and Systems
Abstract: Exoskeletons hold great potential to enhance human locomotion performance, but their development is often hindered by bulky, heavy, and obtrusive actuator and mechanism designs. Here, we present a compact and lightweight hip exoskeleton en- dowed with a custom high torque density actuator and two fold- able mechanisms, namely foldable waist belt with self-alignment mechanism and foldable thigh brace with self-adjusting linear slider mechanism. Our model of actuator electromagnetic design considered four design parameters, including end winding length, stator teeth number, rotor pole pair number and gear ratio that are tailored for portable exoskeletons. Two foldable mechanism enhanced exoskeleton adaptability and user comfort. Benchtop experimental results demonstrated that our actuator can provide an 18 Nm peak torque with a packaging factor improvement of 27% in contrast with state-of-the-art actuators used in exoskeletons. The volume of our exoskeleton was reduced by 55% and the weight was reduced from 3.7kg to 2.7kg compared to our prior design. Preliminary human experiments demonstrated the feasibility of our exoskeleton to reduce metabolic rate during walking and stair climbing for young and older adults.

Keywords: Object Detection, Segmentation and Categorization, Deep Learning Methods, Sensor Fusion
Abstract: Bird's eye view (BEV) segmentation map is a recent development in autonomous driving that provide effective environmental information, such as drivable areas and lane dividers. Most of the existing methods use cameras and LiDAR as inputs for segmentation and the fusion of different modalities is accomplished through either concatenation or addition operations, which fails to exploit fully the correlation and complementarity between modalities. This paper presents the CMGFA (Cross-Modal Group-mix attention Feature Aggregator), an end-to-end learning framework that can adapt to multiple modal feature combinations for BEV segmentation. The CMGFA comprises the following components: (i) The camera has a dual-branch structure that strengthens the linkage between local and global features. (ii) Multi-head deformable cross-attention is applied as cross-modal feature aggregators to aggregate camera, LiDAR, and Radar feature maps in BEV for implicitly fusion. (iii) The Group-Mix attention is used to enrich the attention map feature space and enhance the ability to segment between different categories. We evaluate our proposed method on the nuScenes and Argoverse2 datasets, where the CMGFA significantly outperforms the baseline. The code will be updated at https://github.com/kuangxk2016/CMGFA/tree/master

Keywords: Object Detection, Segmentation and Categorization, Deep Learning for Visual Perception
Abstract: Fisheye cameras, renowned for their panoramic field of view (FOV) of 360 are crucial for surround-view perception in autonomous driving. However, research on object perception in fisheye images lags behind that of standard images. To address this gap, we propose a feature-aligned fisheye object detection network specifically tailored for autonomous driving. Current fisheye perception algorithms often overlook the misalignment issues that typically arise in object detectors. To tackle these challenges in the feature pyramid network (FPN), we introduce a feature-aligned pyramid module (FaPM), which learns pixel transformation offsets to contextually align feature maps. Additionally, we present a location-aligned detection head (LaDH) to align the spatial distribution of classification and regression localization. Integrating these modules into a detection framework results in a novel feature-aligned fisheye object detector. Our method undergoes extensive evaluation on the WoodScape dataset, achieving a mean average precision (mAP) of 32.2%, surpassing the performance of existing methods.

Keywords: Object Detection, Segmentation and Categorization, RGB-D Perception, Deep Learning for Visual Perception
Abstract: Affordance refers to the functional properties that an agent perceives and utilizes from its environment, and is key perceptual information required for robots to perform actions. This information is rich and multimodal in nature. Existing multimodal affordance methods face limitations in extracting useful information, mainly due to simple structural designs, basic fusion methods, and large model parameters, making it difficult to meet the performance requirements for practical deployment. To address these issues, this paper proposes the BiT-Align image-depth-text affordance mapping framework. The framework includes a Bypass Prompt Module (BPM) and a Text Feature Guidance (TFG) attention selection mechanism. BPM integrates the auxiliary modality depth image directly as a prompt to the primary modality RGB image, embedding it into the primary modality encoder without introducing additional encoders. This reduces the model's parameter count and effectively improves functional region localization accuracy. The TFG mechanism guides the selection and enhancement of attention heads in the image encoder using textual features, improving the understanding of affordance characteristics. Experimental results demonstrate that the proposed method achieves significant performance improvements on public AGD20K and HICO-IIF datasets. On the AGD20K dataset, compared with the current state-of-the-art method, we achieve a 6.0% improvement in the KLD metric, while reducing model parameters by 88.8%, demonstrating practical application values. The source code will be made publicly available at https://github.com/DAWDSE/BiT-Align.

Keywords: Object Detection, Segmentation and Categorization, Deep Learning Methods, Sensor Fusion
Abstract: Fusion of LiDAR and RGB data has the potential to enhance outdoor 3D object detection accuracy. To address real-world challenges in outdoor 3D object detection, fusion of LiDAR and RGB input has started gaining traction. However, effective integration of these modalities for precise object detection tasks still remains a largely open problem. To address that, we propose a MultiStream Detection (MuStD) network, which meticulously extracts task-relevant information from both data modalities. The network follows a three-stream structure. Its LiDAR-PillarNet stream extracts sparse 2D pillar features from the LiDAR input while the LiDAR-Height Compression stream computes Bird's-Eye View features. An additional 3D Multimodal stream combines RGB and LiDAR features using UV mapping and polar coordinate indexing. Eventually, the features containing comprehensive spatial, textural, and geometric information are carefully fused and fed to a detection head for 3D object detection. We evaluate our method on the challenging KITTI Object Detection Benchmark using the official testing server. Our approach achieves strong performance, with an average precision (AP) of 85.39% in 3D detection,91.34% in Bird's Eye View (BEV) detection, and 96.39% in 2D detection. These results match or surpass existing state-of-the-art methods. In the difficult ``Hard'' category, our method attains 80.78% AP in 3D detection and 94.04% AP in 2D detection, highlighting its robustness in challenging scenarios. Furthermore, our method runs at 67 ms, demonstrating efficiency and real-time capability. Our code will be released through the MuStD GitHub repository at https://github.com/IbrahimUWA/MuStD.

Keywords: Object Detection, Segmentation and Categorization, Deep Learning for Visual Perception, AI-Based Methods
Abstract: In recent years, capsule robots have gained wide acceptance among doctors and patients for the examination of gastrointestinal diseases due to their non-invasive, safe, and painless advantages. However, the image resolution captured by capsule robots is limited by space size and power, which hinders doctors' ability to accurately assess patients' stomach conditions and real-time control of the capsule robot. This paper proposes the design of two super-resolution networks for capsule robot videos. The first network, EndoVSR, is a high-performance offline video super-resolution network based on a generative adversarial network. It is designed to enhance the resolution of captured videos during offline processing. The second network, Bi-RUN, is a real-time video super-resolution network based on recurrent neural networks. It is designed to enhance the resolution of videos in real-time, enabling doctors to have a clearer view of the stomach condition during the examination. Extensive training and verification of these networks have been conducted using different datasets. All the performance indicators achieved leading positions. Furthermore, simulation experiments were carried out on pig stomachs in vitro to further validate the performance of the proposed networks in practical applications.

Keywords: Object Detection, Segmentation and Categorization, Data Sets for Robotic Vision, Simulation and Animation
Abstract: Vision-based object detectors are a crucial basis for robotics applications as they provide valuable information about object localization in the environment. These need to ensure high reliability in different lighting conditions, occlusions, and visual artifacts, all while running in real-time. Collecting and annotating real-world data for these networks is prohibitively time consuming and costly, especially for custom assets, such as industrial objects, making it untenable for generalization to in-the-wild scenarios. To this end, we present Synthetica, a method for large-scale synthetic data generation for training robust state estimators. This paper focuses on the task of object detection, an important problem which can serve as the front-end for most state estimation problems, such as pose estimation. Leveraging data from a photorealistic ray-tracing renderer, we scale up data generation, generating 2.7 million images, to train highly accurate real-time detection transformers. We present a collection of rendering randomization and training-time data augmentation techniques conducive to robust sim-to-real performance for vision tasks. We demonstrate state-of-the-art performance on the task of object detection while having detectors that run at 50--100Hz which is 9 times faster than the prior SOTA. We further demonstrate the usefulness of our training methodology for robotics applications by showcasing a pipeline for use in the real world with custom objects for which there do not exist prior datasets. Our work highlights the importance of scaling synthetic data generation for robust sim-to-real transfer while achieving the fastest real-time inference speeds. Videos and supplementary information can be found at https://sites.google.com/view/synthetica-vision

Keywords: Object Detection, Segmentation and Categorization, Force and Tactile Sensing, Deep Learning Methods
Abstract: Due to non-destructive testing (NDT) techniques being both expensive and inconvenient in dynamic detection scenarios, innovative alternatives are urgently needed to address cost-efficiency and deployment challenges. We first design TacRoller, a tactile sensor roller for automated characterization of surface defects in composite materials, to address the dilemma. It collects tactile images of defects on the composite's plies by capturing changes caused by deformation of the outer elastomer through the internal camera. It reduces the cost of inspection by 80% to 90% compared to NDT equipment like radiographic testing while ensuring detection efficiency. It takes 58.86 seconds to complete a 35cm x 18cm x 0.5mm dry-woven fabric. Moreover, we collect a total of 2,744 images of samples of dry-woven fabric unidirectional prepreg through TacRoller to form a dataset, including wrinkles, foreign objects and debris (FODs), broken fibre, voids and healthy textures. Subsequently, we propose a multi-order gated aggregation (MOGA)-U-Net to tackle critical challenges of noise sensitivity and multi-scale defect recognition in tactile images, enabling robust segmentation and multi-category classification tasks. The results show that the MOGA-U-Net achieves a test dice coefficient of 76.0% and classification accuracy of 98.9%, outperforming DeepLabV3 and other benchmarks. By providing a scalable and effective NDT substitute, our system realises autonomous defect identification and classification on composite surfaces, thus improving quality control in the production of composites.

Keywords: Object Detection, Segmentation and Categorization, Deep Learning for Visual Perception
Abstract: Robots rely heavily on visual perception to understand and interact with complex environments. To support this capability, modern perception models have become increasingly large and powerful, resulting in high computational costs that hinder their real-time performance in robotic applications. Existing acceleration techniques, such as model pruning and token pruning, focus on reducing architectural or parameter redundancy but still process all object categories, regardless of task requirements. However, in real-world robotic scenarios, different tasks typically require only a subset of object categories. For instance, a service robot may focus on kitchenware while cooking, but shift to furniture and obstacles while cleaning. This task-dependent variation creates opportunities to reduce computational cost by selectively processing relevant information. Existing methods are not designed to exploit this potential for task-specific efficiency. To address this limitation, we propose TaskTP, a task-oriented token pruning method that dynamically adjusts token pruning based on the target category set. A dynamic gating network is introduced between successive Transformer blocks, which evaluates the relevance of each token to the given task. TaskTP allows for more aggressive pruning when fewer categories are required, optimizing computation without sacrificing performance. After a task-agnostic training phase, it can be flexibly configured at deployment time to support any category subset without retraining, making it both efficient and versatile. TaskTP improves the performance of Mask R-CNN from 31.4 fps to 38.5 fps on the COCO dataset. Furthermore, on the ScanNet dataset, where an object search task was defined to simulate real-world robotic applications, processing time was reduced from 3197 ms to 2437 ms, demonstrating significant efficiency gains.

Keywords: RGB-D Perception, Computer Vision for Transportation, Object Detection, Segmentation and Categorization
Abstract: RGB-D has gradually become a crucial data source for understanding complex scenes in assisted driving. However, existing studies have paid insufficient attention to the intrinsic spatial properties of depth maps. This oversight significantly impacts the attention representation, leading to prediction errors caused by attention shift issues. To this end, we propose a novel learnable Depth interaction Pyramid Transformer (DiPFormer) to explore the effectiveness of depth. Firstly, we introduce Depth Spatial-Aware Optimization (Depth SAO) as offset to represent real-world spatial relationships. Secondly, the similarity in the feature space of RGB-D is learned by Depth Linear Cross-Attention (Depth LCA) to clarify spatial differences at the pixel level. Finally, an MLP Decoder is utilized to effectively fuse multi-scale features for meeting real-time requirements. Comprehensive experiments demonstrate that the proposed DiPFormer significantly addresses the issue of attention misalignment in both road detection (+7.5%) and semantic segmentation (+4.9% / +1.5%) tasks. DiPFormer achieves state-of-the-art performance on the KITTI (97.57% F-score on KITTI road and 68.74% mIoU on KITTI-360) and Cityscapes (83.4% mIoU) datasets.

Keywords: Range Sensing, Autonomous Vehicle Navigation
Abstract: Real-time and accurate perception of dynamic objects is crucial for autonomous driving. To better capture the motion information of objects, some methods now employ 4D Doppler point clouds collected by frequency-modulated continuous-wave (FMCW) LiDAR to enhance the detection and tracking of moving objects. Compared to standard time-of-flight (ToF) LiDAR, FMCW LiDAR can provide the relative radial velocity of each point through the Doppler effect, offering a more detailed understanding of an object's motion state. However, despite the proven efficacy of these methods, ablation studies reveal that the direct contribution of Doppler velocity to tracking is limited, with performance gains often resulting from improved object recognition and labeling accuracy.

Revisiting the role of Doppler velocity, this study proposes DopplerTrack, a simple yet effective learning-free tracking method tailored for FMCW LiDAR. DopplerTrack harnesses Doppler velocity for efficient point cloud preprocessing and object detection with O(n) complexity. Furthermore, by exploring the potential motion directions of objects, it reconstructs the full velocity vector, enabling more direct and precise motion prediction. Extensive experiments on four datasets demonstrate that DopplerTrack outperforms existing learning-free and learning-based methods, achieving state-of-the-art tracking performance with strong generalization across diverse scenarios. Moreover, DopplerTrack runs efficiently at 120 Hz on a mobile CPU, making it highly practical for real-world deployment. The code and datasets have been released at https://github. com/12w2/DopplerTrack.

Keywords: Range Sensing, Sensor Fusion, Computer Vision for Automation
Abstract: Scene view synthesis, which generates novel views from limited perspectives, is increasingly vital for applications like virtual reality, augmented reality, and robotics. Unlike object-based tasks, such as generating 360° views of a car, scene view synthesis handles entire environments where non-uniform observations pose unique challenges for stable rendering quality. To address this issue, we propose a novel approach: renderability field-guided gaussian splatting (RF-GS). This method quantifies input inhomogeneity through a renderability field, guiding pseudo-view sampling to enhanced visual consistency. To ensure the quality of wide-baseline pseudo-views, we train an image restoration model to map point projections to visible-light styles. Additionally, our validated hybrid data optimization strategy effectively fuses information of pseudo-view angles and source view textures. Comparative experiments on simulated and real-world data show that our method outperforms existing approaches in rendering stability.

Keywords: Range Sensing, RGB-D Perception, Deep Learning for Visual Perception
Abstract: The recent development of foundation models for monocular depth estimation such as Depth Anything paved the way to zero-shot monocular depth estimation. Since it returns an affine-invariant disparity map, the favored technique to recover the metric depth consists in fine-tuning the model. However, this stage is not straightforward, it can be costly and time-consuming because of the training and the creation of the dataset. The latter must contain images captured by the camera that will be used at test time and the corresponding ground truth. Moreover, the fine-tuning may also degrade the generalizing capacity of the original model. Instead, we propose in this paper a new method to rescale Depth Anything predictions using 3D points provided by sensors or techniques such as low-resolution LiDAR or structure-from-motion with poses given by an IMU. This approach avoids fine-tuning and preserves the generalizing power of the original depth estimation model while being robust to the noise of the sparse depth, of the camera-LiDAR calibration or of the depth model. Our experiments highlight enhancements relative to zero-shot monocular metric depth estimation methods, competitive results compared to fine-tuned approaches and a better robustness than depth completion approaches. Code available at github.com/ENSTA-U2IS-AI/depth-rescaling.

Keywords: AI-Based Methods, Reactive and Sensor-Based Planning, Force and Tactile Sensing
Abstract: Electrical capacitance tomography (ECT) is a contactless and non-invasive imaging technique, which visualizes the internal permittivity distribution around a region utilizing boundary capacitance measurements. It has been widely used in the fields of object classification, tactile sensing and multiphase flows monitoring. However, due to the inherent non-linearity and ill-conditioned nature of the ECT inverse problem, its practical implementation remains limited by challenges in the image reconstruction accuracy. To tackle the above problems, we propose a patch-based transformer method (PT) for an accurate reconstruction of ECT images. Specifically, the complex capacitance-to-image mapping is systematically decoupled into the capacitance-to-patch feature extraction and patch-to-image reconstruction, enabling more efficient and accurate permittivity distribution recovery through localized feature learning and global context integration. Additionally, a simulation ECT dataset for objects with varying sizes and positions is established.

Keywords: Range Sensing, Data Sets for Robotic Vision
Abstract: Depth from focus (DFF) is a well-established method for measuring depth in vision systems. However, its efficacy and accuracy are limited by the slow speed required to capture high-quality focal stacks. We address this limitation by leveraging emerging hardware technologies: the event camera and liquid lens. In this paper, we introduce an innovative approach called Event-Based Depth from Focus (EDFF). We present a prototype system and propose Event Cancellation Score (ECS) as a novel metric to efficiently detect event data focus. To validate the effectiveness of our system, we have curated the first EDFF dataset, which comprises event recordings of focal sweeps performed on 3D-printed test targets. Comparative analysis against existing event focus detection algorithms demonstrates the superior performance of our algorithm in the EDFF task.

Keywords: Range Sensing, SLAM, Mapping
Abstract: In the deep learning era, 2D LiDAR perception is often overlooked as research prioritizes 3D point clouds. Yet, 2D LiDAR remains essential for low-cost robotic systems due to its affordability. Despite its simplicity, it faces major challenges-not only in perception but also in learning itself, as data sparsity and limited features hinder effective framework development. Additionally, fundamental structural differences prevent direct adaptation of 3D perception networks. To address these aforementioned concerns, we proposes LSW-Net(Laser Scan Weight-Net), a self-supervised framework for 2D LiDAR perception, enabling end-to-end learning from raw point clouds to adaptive weight extraction. This framework provides a scalable and lightweight solution for 2d LiDAR environmental perception, and its self-supervised nature reduces annotation costs.It includes: (i) a general 2D Laser Encoder (LS-Encoder) that integrates local convolutional perception with global attention perception to extract multi-scale spatiotemporal features; and (ii) an interpretable weight extraction module (Weight Extractor) that dynamically quantifies the contribution of each point in environmental perception tasks through contrastive learning and physical consistency constraints. Evaluated on diverse scenes for point cloud registration and SLAM tasks, LSW-Net outperforms traditional methods in feature discriminability and adaptability.Additionally, we performed solid ablation experiments to substantiate the rationality of our approach. Our code is available at https://github.com/YukiaCUI/LSW-Net.

Keywords: Range Sensing, Mapping, Field Robots
Abstract: Ground point cloud extraction is crucial for route planning of autonomous vehicles in unstructured environments. However, mainstream point cloud extraction methods are susceptible to inaccuracies due to the indistinct obstacle-ground boundary. Furthermore, addressing uneven feature distribution usually necessitates region segmentation, which increases computational demands. To achieve a balance between efficiency and accuracy in ground extraction, we propose a two-stage framework based on adaptive bin partition and grid projection. Firstly, the point cloud is divided into bins based on point cloud distribution and then projected onto the uniform grids, ensuring robust consideration of uneven features. Subsequently, the grid-based coarse extraction is performed by analyzing the grid characteristics to enable a rapid preliminary extraction. Furthermore, the coarse results are reprojected into point cloud form, and the ground-obstacle boundary regions are finally refined by incorporating elevation and curvature probabilities. Evaluations on RELLIS-3D dataset and field tests conducted across typical unstructured scenarios demonstrate that the proposed method achieves promising extraction performance compared to state-of-the-art methods.

Keywords: Reinforcement Learning, Whole-Body Motion Planning and Control, Deep Learning Methods
Abstract: The whole-body control of mobile manipulators for the 6-DOF trajectory following task is the basis of many continuous tasks. However, traditional control strategies rely on accurate models and expert knowledge for solving the trajectory following task. Deep reinforcement learning (DRL) provides a promising model-free solution, but it is sample-inefficient. To this end, we propose Trajectory Following Hindsight Experience Replay (TF-HER), a sample-efficient DRL algorithm for the whole-body coupled trajectory following task with dense rewards. TF-HER builds a multi-trajectory state space, and relabels the low-reward data to generate informative high-reward experiences. Also, the distributional shift caused by the relabeling is corrected by estimating the density ratio of relabeled experiences. Extensive demonstrations on both nonholonomic and holonomic bases in simulation validate that our algorithm can accelerate the model convergence and significantly improve the sample efficiency. Furthermore, we present real-world experiments to demonstrate the effectiveness of our approach. The code is available: https://github.com/IRMV-Manipulation-Group/TF-HER

Keywords: Reinforcement Learning, Collision Avoidance, Motion and Path Planning
Abstract: We explore the problem of path planning for mobile robots in unknown environments using deep reinforcement learning (DRL). This paper proposes a multi-layer long short-term memory twin delayed deep deterministic policy gradient (MLTD3) algorithm for unknown environments. First, we introduce multi-layer long short-term memory (LSTM) networks to the actor network of the twin delayed deep deterministic policy gradient (TD3), to capture rich historical information to alleviate local optimal solutions in local path planning. Secondly, Poisson coding is employed in the state space to deal with the uncertainty of environments, so that the algorithm can discern the relationship within long sequence information. Then novel extrinsic and intrinsic reward functions are designed to avoid the dynamic obstacles in environments. Finally, the performance of the proposed algorithm is verified through simulations in ROS Gazebo and physical experiments in an unknown environment.

Keywords: Reinforcement Learning, Imitation Learning, Deep Learning Methods
Abstract: Human guidance has emerged as a powerful tool for enhancing reinforcement learning (RL). However, conventional forms of guidance such as demonstrations or binary scalar feedback can be challenging to collect or have low information content, motivating the exploration of other forms of human input. Among these, relative feedback (i.e., feedback on how to improve an action, such as “more to the left”) offers a good balance between usability and information richness. Previous research has shown that relative feedback can be used to enhance policy search methods. However, these efforts have been limited to specific policy classes and use feedback inefficiently. In this work, we introduce a novel method to learn from relative feedback and combine it with off-policy reinforcement learning. Through evaluations on two sparse-reward tasks, we demonstrate our method can be used to improve the sample efficiency of reinforcement learning by guiding its exploration process. Additionally, we show it can adapt a policy to changes in the environment or the user's preferences. Finally, we demonstrate real-world applicability by employing our approach to learn a navigation policy in a sparse reward setting.

Keywords: Reinforcement Learning, Learning from Demonstration, Intelligent Transportation Systems
Abstract:   Autonomous driving with reinforcement learning (RL) has significant potential. However, applying RL in real-world settings remains challenging due to the need for safe, efficient, and robust learning. Incorporating human expertise into the learning process can help overcome these challenges by reducing risky exploration and improving sample efficiency. In this work, we propose a reward-free, active human-in-the-loop learning method called Human-Guided Distributional Soft Actor-Critic (H-DSAC). Our method combines Proxy Value Propagation (PVP) and Distributional Soft Actor-Critic (DSAC) to enable efficient and safe training in real-world environments. The key innovation is the construction of a distributed proxy value function within the DSAC framework. This function encodes human intent by assigning higher expected returns to expert demonstrations and penalizing actions that require human intervention. By extrapolating these labels to unlabeled states, the policy is effectively guided toward expert-like behavior. With a well-designed state space, our method achieves real-world driving policy learning within practical training times. Results from both simulation and real-world experiments demonstrate that our framework enables safe, robust, and sample-efficient learning for autonomous driving.

Keywords: Reinforcement Learning, Dexterous Manipulation, Deep Learning in Grasping and Manipulation
Abstract: Autonomous learning of dexterous, long-horizon robotic skills has been a longstanding pursuit of embodied AI. Recent advances in robotic reinforcement learning (RL) have demonstrated remarkable performance and robustness in real-world visuomotor control tasks. However, applying RL in the real world faces challenges such as low sample efficiency, slow exploration, and significant reliance on human intervention. In contrast, simulators offer a safe and efficient environment for extensive exploration and data collection, while the visual sim-to-real gap, often a limiting factor, can be mitigated using real-to-sim techniques. Building on these, we propose SimLauncher, a novel framework that combines the strengths of real-world RL and real-to-sim-to-real approaches to overcome these challenges. Specifically, we first pre-train policies in digital twin simulation environments, which then benefit real-world RL in two ways: (1) bootstrapping target values using real-world demonstrations derived from simulation policy rollouts and extensive simulation demonstrations, and (2) accelerating exploration using the pre-trained policy. We conduct comprehensive experiments across long-horizon, contact-rich, and dexterous hand manipulation tasks. Compared to prior real-world RL approaches, SimLauncher significantly improves training efficiency and achieves near-perfect success rates. We hope this work serves as a proof-of-concept and inspires further research on leveraging large-scale simulation pre-training to benefit real-world robotic learning.

Keywords: Reinforcement Learning, Vision-Based Navigation, Aerial Systems: Perception and Autonomy
Abstract: Autonomous visual navigation is an essential element in robot autonomy. Reinforcement learning (RL) offers a promising policy training paradigm. However, existing RL methods suffer from high sample complexity, poor sim-to-real transfer, and limited runtime adaptability. These problems are particularly challenging for drones, with complex nonlinear and unstable dynamics, and strong dynamic coupling between control and perception. In this paper, we propose a novel framework that integrates 3D Gaussian Splatting (3DGS) with differentiable deep reinforcement learning (DDRL) to train vision-based drone navigation policies. By leveraging high-fidelity 3D scene representations and differentiable simulation, our method improves sample efficiency and sim-to-real transfer. Additionally, we incorporate a Context-aided Estimator Network (CENet) to adapt to environmental variations at runtime. Moreover, by curriculum training in a mixture of different surrounding environments, we achieve in-task generalization, the ability to solve new instances of a task not seen during training. Drone hardware experiments demonstrate our method's high training efficiency compared to state-of-the-art RL methods, zero shot sim-to-real transfer for real robot deployment without fine tuning, and ability to adapt to new instances within the same task class (e.g. to fly through a gate at different locations with different distractors in the environment). Our simulator and training framework are open-sourced at: https://github.com/Qianzhong-Chen/grad_nav.

Keywords: Reinforcement Learning, Aerial Systems: Mechanics and Control
Abstract: Quadcopter attitude control involves two tasks: smooth attitude tracking and aggressive stabilization from arbitrary states. Although both can be formulated as tracking problems, their distinct state spaces and control strategies complicate a unified reward function. We propose a multitask deep reinforcement learning framework that leverages parallel simulation with IsaacGym and a Graph Convolutional Network (GCN) policy to address both tasks effectively. Our multitask Soft Actor-Critic (SAC) approach achieves faster, more reliable learning and higher sample efficiency than single-task methods. We validate its real-world applicability by deploying the learned policy—a compact two-layer network with 24 neurons per layer—on a Pixhawk flight controller, achieving 400 Hz control without extra computational resources. We provide our code at url{https://github.com/robot-perception-group/GraphMTSAC_U AV/}.

Keywords: Reinforcement Learning, Integrated Planning and Learning
Abstract: Symbolic task representation is a powerful tool for encoding human instructions and domain knowledge. Such instructions guide robots to accomplish diverse objectives and meet constraints through reinforcement learning (RL). Most existing methods are based on fixed mappings from environmental states to symbols. However, in inspection tasks, where equipment conditions must be evaluated from multiple perspectives to avoid errors of oversight, robots must fulfill the same symbol from different states. To help robots respond to flexible symbol mapping, we propose representing symbols and their mapping specifications separately within an RL policy. This approach imposes on RL policy to learn combinations of symbolic instructions and mapping specifications, requiring an efficient learning framework. To cope with this issue, we introduce an approach for learning flexible policies called Symbolic Instructions with Adjustable Mapping Specifications (SIAMS). This paper represents symbolic instructions using linear temporal logic (LTL), a formal language that can be easily integrated into RL. Our method addresses the diversified completion patterns of instructions by (1) a specification-aware state modulation, which embeds differences in mapping specifications in state features, and (2) a symbol-number-based task curriculum, which gradually provides tasks according to the learning’s progress. Evaluations in 3D simulations with discrete and continuous action spaces demonstrate that our method outperforms context-aware multitask RL comparisons.

Keywords: Imitation Learning, Deep Learning in Grasping and Manipulation, Learning from Demonstration
Abstract: In imitation learning for robotic manipulation, decomposing object manipulation tasks into sub-tasks enables the reuse of learned skills and the combination of learned behaviors to perform novel tasks, rather than simply replicating demonstrated motions. Human gaze is closely linked to hand movements during object manipulation. We hypothesize that an imitating agent's gaze control—fixating on specific landmarks and transitioning between them—simultaneously segments demonstrated manipulations into sub-tasks. This study proposes a simple yet robust task decomposition method based on gaze transitions. Using teleoperation, a common modality in robotic manipulation for collecting demonstrations, in which a human operator's gaze is measured and used for task decomposition as a substitute for an imitating agent's gaze. Our approach ensures consistent task decomposition across all demonstrations for each task, which is desirable in contexts such as machine learning. We evaluated the method across demonstrations of various tasks, assessing the characteristics and consistency of the resulting sub-tasks. Furthermore, extensive testing across different hyperparameter settings confirmed its robustness, making it adaptable to diverse robotic systems. Our code is available at htt ps://github.com/crumbyRobotics/GazeTaskDecomp.

Keywords: Imitation Learning, Learning from Demonstration
Abstract: Imitation learning (IL) aims to enable robots to perform tasks autonomously by observing a few human demonstrations. Recently, a variant of IL, called In-Context IL, utilized off-the-shelf large language models (LLMs) as instant policies that understand the context from a few given demonstrations to perform a new task, rather than explicitly updating network models with large-scale demonstrations. However, its reliability in the robotics domain is undermined by hallucination issues such as LLM-based instant policy, which occasionally generates poor trajectories that deviate from the given demonstrations. To alleviate this problem, we propose a new robust in-context imitation learning algorithm called the robust instant policy (RIP), which utilizes a Student’s t-regression model to be robust against the hallucinated trajectories of instant policies to allow reliable trajectory generation. Specifically, RIP generates several candidate robot trajectories to complete a given task from an LLM and aggregates them using the Student's t-distribution, which is beneficial for ignoring outliers (i.e., hallucinations); thereby, a robust trajectory against hallucinations is generated. Our experiments, conducted in both simulated and real-world environments, show that RIP significantly outperforms state-of-the-art IL methods, with at least 26% improvement in task success rates, particularly in low-data scenarios for everyday tasks. Video results available at https://sites.google.com/view/robustinstantpolicy.

Keywords: Imitation Learning, Learning from Demonstration, Reinforcement Learning
Abstract: Surgical robot task automation has recently attracted great attention due to its potential to benefit both surgeons and patients. Reinforcement learning (RL) based approaches have demonstrated promising ability to perform automated surgical manipulations on various tasks. To address the exploration challenge, expert demonstrations can be utilized to enhance the learning efficiency via imitation learning (IL) approaches. However, the successes of such methods normally rely on both states and action labels. Unfortunately, action labels can be hard to capture or their manual annotation is prohibitively expensive owing to the requirement for expert knowledge. Emulating expert behaviour using noisy or inaccurate labels poses significant risks, including unintended surgical errors that may result in patient discomfort or, in more severe cases, tissue damage. It therefore remains an appealing and open problem to leverage expert data composed of pure states into RL. In this work, we present an actor-critic RL framework, termed AC-SSIL, to overcome this challenge of improving learning process with state-only demonstrations collected by an unknown expert policy. It adopts a self-supervised IL method, dubbed SSIL, to effectively incorporate expert states into RL paradigms by retrieving from demonstrations the nearest neighbours of the query state and utilizing the bootstrapping of actor networks. It applies similarity-based regularization and improves its prediction capacity jointly with the actor network. We showcase through experiments on an open-source surgical simulation platform that our method delivers remarkable improvements over the RL baseline and exhibits comparable performance against action based IL methods, which implies the efficacy and potential of our method for expert demonstration-guided learning scenarios. Code will be made publicly available at https://github.com/Jingshuai-cqu/AC-SSIL.

Keywords: Imitation Learning, Multi-Robot Systems, Learning from Demonstration
Abstract: In robotic systems, the performance of reinforcement learning depends on the rationality of predefined reward functions. However, manually designed reward functions often lead to policy failures due to inaccuracies. Inverse Reinforcement Learning (IRL) addresses this problem by inferring implicit reward functions from expert demonstrations. Nevertheless, existing methods rely heavily on large amounts of expert demonstrations to accurately recover the reward function. The high cost of collecting expert demonstrations in robotic applications, particularly in multi-robot systems, severely hinders the practical deployment of IRL. Consequently, improving sample efficiency has emerged as a critical challenge in multi-agent inverse reinforcement learning (MIRL). Inspired by the symmetry inherent in multi-agent systems, this work theoretically demonstrates that leveraging symmetry enables the recovery of more accurate reward functions. Building upon this insight, we propose a universal framework that integrates symmetry into existing multi-agent adversarial IRL algorithms, thereby significantly enhancing sample efficiency. Experimental results from multiple challenging tasks have demonstrated the effectiveness of this framework. Further validation in physical multi-robot systems has shown the practicality of our method.

Keywords: Learning from Demonstration, Imitation Learning, Human-Robot Collaboration
Abstract: Imitation learning enables robots to learn new tasks from human examples. One fundamental limitation while learning from humans is causal confusion. Causal confusion occurs when the robot's observations include both task-relevant and extraneous information: for instance, a robot's camera might see not only the intended goal, but also clutter and changes in lighting within its environment. Because the robot does not know which aspects of its observations are important a priori, it often misinterprets the human's examples and fails to learn the desired task. To address this issue, we highlight that --- while the robot learner may not know what to focus on --- the human teacher does. In this paper we propose that the human proactively marks key parts of their task with small, lightweight beacons. Under our framework (RECON) the human attaches these beacons to task-relevant objects before providing demonstrations: as the human shows examples of the task, beacons track the position of marked objects. We then harness this offline beacon data to train a task-relevant state embedding. Specifically, we embed the robot's observations to a latent state that is correlated with the measured beacon readings: in practice, this causes the robot to autonomously filter out extraneous observations and make decisions based on features learned from the beacon data. Our simulations and a real robot experiment suggest that this framework for human-placed beacons mitigates causal confusion. Indeed, we find that using RECON significantly reduces the number of demonstrations needed to convey the task, lowering the overall time required for human teaching.

Keywords: Imitation Learning, Bioinspired Robot Learning, Representation Learning
Abstract: Training robots to operate effectively in environments with uncertain states—such as ambiguous object properties or unpredictable interactions—remains a longstanding challenge in robotics. Imitation learning methods typically rely on successful examples and often neglect failure scenarios where uncertainty is most pronounced. To address this limitation, we propose the Uncertainty-driven Foresight Recurrent Neural Network (UF-RNN), a model that combines standard time-series prediction with an active “Foresight” module. This module performs internal simulations of multiple future trajectories and refines the hidden state to minimize predicted variance, enabling the model to selectively explore actions under high uncertainty. We evaluate UF-RNN on a door-opening task in both simulation and a real-robot setting, demonstrating that, despite the absence of explicit failure demonstrations, the model exhibits robust adaptation by leveraging self-induced chaotic dynamics in its latent space. When guided by the Foresight module, these chaotic properties stimulate exploratory behaviors precisely when the environment is ambiguous, yielding improved success rates compared to conventional stochastic RNN baselines. These findings suggest that integrating uncertainty-driven foresight into imitation learning pipelines can significantly enhance a robot’s ability to handle unpredictable real-world conditions.

Keywords: Imitation Learning, Reinforcement Learning, Manipulation Planning
Abstract: Reward engineering for policy learning has been a long-standing challenge in robotics. Recently, to avoid manual reward designs, vision-language models (VLMs) have shown promise in defining rewards for teaching robots manipulation skills. However, existing work often provides reward guidance that is too coarse, leading to insufficient learning processes. In this paper, we address this issue by implementing more fine-grained reward guidance. Specifically, we decompose tasks into simpler sub-tasks, using this decomposition to offer more informative reward guidance with VLMs. We also propose a VLM-based self imitation learning process to speed up learning. Empirical evidence demonstrates that our algorithm consistently outperforms baselines such as CLIP, LIV, and RoboCLIP. Specifically, our algorithm achieves a 5.4 times higher average success rates compared to the best baseline, RoboCLIP, across a series of manipulation tasks.

Keywords: Dual Arm Manipulation, Imitation Learning, Physically Assistive Devices
Abstract: Caregiving is a vital role for domestic robots, especially the repositioning care has immense societal value, critically improving the health and quality of life of individuals with limited mobility. However, repositioning task is a challenging area of research, as it requires robots to adapt their motions while interacting flexibly with patients. The task involves several key challenges: (1) applying appropriate force to specific target areas; (2) performing multiple actions seamlessly, each requiring different force application policies; and (3) adaptation under uncertain positional conditions. To address these, we propose a deep neural network (DNN)-based architecture utilizing proprioceptive and visual attention mechanisms, along with impedance control to regulate the robot's movements. Using the dual-arm humanoid robot Dry-AIREC, the proposed model successfully generated motions to insert the robot's hand between the bed and a mannequin's back without applying excessive force, and it supported the transition from a supine to a lifted-up position.

Keywords: Deep Learning for Visual Perception, Deep Learning Methods, RGB-D Perception
Abstract: Three-dimensional local descriptors are crucial for encoding geometric surface properties, making them essential for various point cloud understanding tasks. Among these descriptors, GeDi has demonstrated strong zero-shot 6D pose estimation capabilities but remains computationally impractical for real-world applications due to its expensive inference process. Can we retain GeDi's effectiveness, while significantly improving its efficiency? In this paper, we explore this question by introducing a knowledge distillation framework that trains an efficient student model to regress local descriptors from a GeDi teacher. Our key contributions include: an efficient large-scale training procedure that ensures robustness to occlusions and partial observations while operating under compute and storage constraints, and a novel loss formulation that handles weak supervision from non-distinctive teacher descriptors. We validate our approach on five BOP Benchmark datasets and demonstrate a significant reduction in inference time while maintaining competitive performance with existing methods, bringing zero-shot 6D pose estimation closer to real-time feasibility.

Keywords: Deep Learning for Visual Perception, Object Detection, Segmentation and Categorization, Semantic Scene Understanding
Abstract: Adversarial patch defense has made significant progress recently, but defending against natural-looking patches remains a challenge due to their content-agnostic nature. We hypothesize that these patches exhibit position-related anomalies and are inspired by anomaly detection techniques.

However, directly applying existing anomaly detection methods to patch detection behaves poorly because patches are a subset of anomalies, and using general anomalous features may introduce irrelevant anomalies. Additionally, anomaly detection datasets are primarily derived from industrial scenes, leading to out-of-distribution issues. To address these challenges, we propose a patch-agnostic defense method based on anomaly knowledge learning. It fine-tunes the Segment Anything Model in a self-supervised manner using an anomaly dataset, enabling the model’s image encoder to generate embeddings with enhanced activation for anomalous regions.It also designs a Cross Attention Patch Decoder based on cross-modal attention mechanisms to compute the mutual information between patch prediction probability maps and anomaly activation maps for patch localization. Experimental results on public datasets demonstrate the effectiveness of the proposed method.

Keywords: Deep Learning for Visual Perception, Object Detection, Segmentation and Categorization, Intelligent Transportation Systems
Abstract: Efficiency is quite important for 3D lane detection while previous detectors are either computationally expensive or difficult for optimization. To bridge this gap, we propose a fully convolutional detector named GroupLane, which is simple, fast, and still maintains high detection precision. Specifically, we first propose to split extracted feature into multiple groups along the channel dimension and employ every group to represent a prediction. In this way, GroupLane realizes end-to-end detection like DETR based on pure convolutional neural networks. Then, we propose to represent lanes by performing row-wise classification in bird’s eye view and devise a set of detection heads. Compared with existing row-wise classification implementations that only support recognizing vertical lanes, ours can detect both vertical and horizontal ones. Additionally, a matching algorithm named single-win one-to-one matching is developed to associate predictions with labels during training. Extensive experiments are conducted to verify the effectiveness of the proposed strategies, and the results suggest that GroupLane achieves the best performance with high inference speed on both the popular OpenLane and Once-3DLanes benchmarks. In addition, GroupLane is the first fully convolutional 3D lane detector that achieves end-to-end detection without post-processing.

Keywords: Deep Learning for Visual Perception, Computer Vision for Automation
Abstract: Recent advancements in autonomous driving (AD) systems have highlighted the potential of world models in achieving robust and generalizable performance across both ordinary and challenging driving conditions. However, a key challenge remains: precise and flexible camera pose control, which is crucial for accurate viewpoint transformation and realistic simulation of scene dynamics. In this paper, we introduce PosePilot, a lightweight yet powerful framework that significantly enhances camera pose controllability in generative world models. Drawing inspiration from self-supervised depth estimation, PosePilot leverages structure-from-motion principles to establish a tight coupling between camera pose and video generation. Specifically, we incorporate self-supervised depth and ego-motion readouts, allowing the model to infer depth and relative camera motion directly from video sequences. These outputs drive pose-aware frame warping, guided by a photometric warping loss that enforces geometric consistency across synthesized frames. To further refine camera pose estimation, we introduce a reverse warping step and a pose regression loss, improving viewpoint precision and adaptability. Extensive experiments on autonomous driving and general-domain video datasets demonstrate that PosePilot significantly enhances structural understanding and motion reasoning in both diffusion-based and auto-regressive world models. By steering camera pose with self-supervised depth, PosePilot sets a new benchmark for pose controllability, enabling physically consistent, reliable viewpoint synthesis in generative world models.

Keywords: Deep Learning for Visual Perception, Surveillance Robotic Systems, Computer Vision for Transportation
Abstract: Detecting driver fatigue is critical for road safety, as drowsy driving remains a leading cause of traffic accidents. Many existing solutions rely on computationally demanding deep learning models, which result in high latency and are unsuitable for embedded robotic devices with limited resources (such as intelligent vehicles/cars) where rapid detection is necessary to prevent accidents. This paper introduces LiteFat, a lightweight spatio-temporal graph learning model designed to detect driver fatigue efficiently while maintaining high accuracy and low computational demands. LiteFat involves converting streaming video data into spatio-temporal graphs (STG) using facial landmark detection, which focuses on key motion patterns and reduces unnecessary data processing. LiteFat uses MobileNet to extract facial features and create a feature matrix for the STG. A lightweight spatio-temporal graph neural network is then employed to identify signs of fatigue with minimal processing and low latency. Experimental results on benchmark datasets show that LiteFat performs competitively while significantly reduced computational complexity and latency compared to the state-of-the-art methods. This work advances the development of real-time, resource-efficient human fatigue detection systems that can be implemented upon embedded robotic devices.

Keywords: Deep Learning for Visual Perception, Deep Learning Methods, Semantic Scene Understanding
Abstract: Deploying real-time spatial perception on edge devices requires efficient multi-task models that leverage complementary task information while minimizing computational overhead. In this paper, we introduce Multi-Mono-Hydra (M2H), a novel multi-task learning framework designed for semantic segmentation and depth, edge, and surface normal estimation from a single monocular image. Unlike conventional approaches that rely on independent single-task models or shared encoder-decoder architectures, M2H introduces a Window-Based Cross-Task Attention Module that enables structured feature exchange while preserving task-specific details, improving prediction consistency across tasks. Built on a lightweight ViT-based DINOv2 backbone, M2H is optimized for real-time deployment and serves as the backbone for monocular spatial perception systems, a framework for 3D scene graph construction in dynamic environments. Comprehensive evaluations demonstrate that M2H outperforms state-of-the-art (SOTA) multi-task models on NYUDv2, exceeds single-task depth and semantic baselines on Hypersim, and achieves superior performance on Cityscapes datasets, all while maintaining computational efficiency on laptop hardware. Beyond curated benchmarks, we validate M2H on real-world data, demonstrating its practicality in spatial perception tasks. We provide our implementation and pretrained models at https://github.com/UAV-Centre-ITC/M2H.git.

Keywords: Deep Learning for Visual Perception, AI-Based Methods, Data Sets for Robotic Vision
Abstract: Change detection has long been used for various tasks. With advancements in robotic systems and computer vision, change detection techniques can be further explored for diverse applications. Current state-of-the-art methods primarily use either satellite images or street-level images to detect changes. However, the techniques used for these two types of images differ substantially, though their core objective remains identical.

We introduce a spectral-temporal attention network capable of detecting changes in both satellite and street-level images across multiple temporal instances. Additionally, we present an indoor environmental dataset featuring significantly more frequent changes. We analyze the impact of temporal and spatial domain shifts on the performance of various methods and demonstrate that performing attention in the spectral domain not only enhances overall performance but also increases robustness against spatial domain shifts.

Keywords: Deep Learning for Visual Perception, Perception for Grasping and Manipulation, Deep Learning in Grasping and Manipulation
Abstract: Localizing predefined 3D keypoints in a 2D image is an effective way to establish 3D-2D correspondences for instance-level 6DoF object pose estimation. However, unreliable localization results of invisible keypoints degrade the quality of correspondences. In this paper, we address this issue by localizing the important keypoints in terms of visibility. Since keypoint visibility information is currently missing in the dataset collection process, we propose an efficient way to generate binary visibility labels from available object-level annotations, for keypoints of both asymmetric objects and symmetric objects. We further derive real-valued visibility-aware importance from binary labels based on the PageRank algorithm. Taking advantage of the flexibility of our visibility-aware importance, we construct VAPO (Visibility-Aware POse estimator) by integrating the visibility-aware importance with a state-of-the-art pose estimation algorithm, along with additional positional encoding. VAPO can work in both CAD-based and CAD-free settings. Extensive experiments are conducted on popular pose estimation benchmarks including Linemod, Linemod-Occlusion, and YCB-V, demonstrating that VAPO clearly achieves state-of-the-art performances. Project page: https://github.com/RuyiLian/VAPO.

Keywords: Human Factors and Human-in-the-Loop, Learning from Demonstration
Abstract: Previous methods for Learning from Demonstration leverage several approaches for a human to teach motions to a robot, including teleoperation, kinesthetic teaching, and natural demonstrations. However, little previous work has explored more general interfaces that allow for multiple demonstration types. Given the varied preferences of human demonstrators and task characteristics, a flexible tool that enables multiple demonstration types could be crucial for broader robot skill training. In this work, we propose Versatile Demonstration Interface (VDI), an attachment for collaborative robots that simplifies the collection of three common types of demonstrations. Designed for flexible deployment in industrial settings, our tool requires no additional instrumentation of the environment. Our prototype interface captures human demonstrations through a combination of vision, force sensing, and state tracking (e.g., through the robot proprioception or AprilTag tracking). Through a user study where we deployed our prototype VDI at a local manufacturing innovation center with manufacturing experts, we demonstrated VDI in representative industrial tasks. Interactions from our study highlight the practical value of VDI's varied demonstration types, expose a range of industrial use cases for VDI, and provide insights for future tool design.

Keywords: Imitation Learning, AI-Enabled Robotics, Learning from Demonstration
Abstract: Imitation learning frameworks for robotic manipulation have drawn attention in the recent development of language model grounded robotics. However, the success of the frameworks largely depends on the coverage of the demonstration cases: When the demonstration set does not include examples of how to act in all possible situations, the action may fail and can result in cascading errors. To solve this problem, we propose a framework that uses serialized Finite State Machine (FSM) to generate demonstrations and improve the success rate in manipulation tasks requiring a long sequence of precise interactions. To validate its effectiveness, we use environmentally evolving and long-horizon puzzles that require long sequential actions. Experimental results show that our approach achieves a success rate of up to 98% in these tasks, compared to the controlled condition using existing approaches, which only had a success rate of up to 60%, and, in some tasks, almost failed completely. The source code for this project can be accessed at https://imitate.finite-state.com/.

Keywords: Imitation Learning, Learning from Demonstration, Reinforcement Learning
Abstract: We study policy distillation under privileged information, where a student policy with only partial observations must learn from a teacher with full-state access. A key challenge is information asymmetry: the student cannot directly access the teacher’s state space, leading to distributional shifts and policy degradation. Existing approaches either modify the teacher to produce realizable but sub-optimal demonstrations or rely on the student to explore missing information independently, both of which are inefficient. Our key insight is that the student should strategically interact with the teacher querying only when necessary and resetting from recovery states to stay on a recoverable path within its own observation space. We introduce two methods: (i) an imitation learning approach that adaptively determines when the student should query the teacher for corrections, and (ii) a reinforcement learning approach that selects where to initialize training for efficient exploration. We validate our methods in both simulated and real-world robotic tasks, demonstrating significant improvements over standard teacher-student baselines in training efficiency and final performance.

Keywords: Learning from Demonstration, Motion and Path Planning, Datasets for Human Motion
Abstract: In robotics, Learning from Demonstration (LfD) aims to transfer skills to robots by using multiple demonstrations of the same task. These demonstrations are recorded and processed to extract a consistent skill representation. This process typically requires temporal alignment through techniques such as Dynamic Time Warping (DTW). In this paper, we consider a novel algorithm, named Spatial Sampling (SS), specifically designed for robot trajectories, that enables time-independent alignment of the trajectories by providing an arc-length parametrization of the signals. This approach eliminates the need for temporal alignment, enhancing the accuracy and robustness of skill representation, especially when recorded movements are subject to intermittent motions or extremely variable speeds, a common characteristic of operations based on kinesthetic teaching, where the operator may encounter difficulties in guiding the end-effector smoothly. To prove this, we built a custom publicly available dataset of robot recordings to test real-world movements, where the user tracks the same geometric path multiple times, with motion laws that vary greatly and are subject to starting and stopping. The SS demonstrates better performances against state-of-the-art algorithms in terms of (i) trajectory synchronization and (ii) quality of the extracted skill.

Keywords: Learning from Demonstration, Imitation Learning, Deep Learning in Grasping and Manipulation
Abstract: Learning robust and generalizable manipulation skills from few demonstrations remains a key challenge in robotics, with broad applications in industrial automation and service robotics. While recent imitation learning methods have achieved impressive results, they often require large amounts of demonstration data and struggle to generalize across different spatial variants. In this work, we propose a framework that learns 3D manipulation policies from only 10 demonstrations while achieving robust generalization to unseen spatial configurations. Our framework consists of two key modules: a Semantic Guided Perception module that extracts task-aware 3D representations from RGB-D inputs using semantic priors, and a Spatial Generalized Decision module implementing a diffusion-based policy that preserves spatial equivariance through denoising. Central to our framework is a spatially equivariant training strategy, which adapts 2D data augmentation principles to 3D manipulation by maintaining gripper-object spatial relationships during trajectory augmentation. We validate our framework through extensive experiments on both simulation benchmarks and real-world robotic systems. Our method demonstrates a 60–70% improvement in success rates over state-of-the-art approaches on a series of challenging tasks, particularly under significant object pose variations. This work shows significant potential for advancing efficient, generalizable manipulation skill learning in real-world applications.

Keywords: Learning from Demonstration, Imitation Learning
Abstract: In contrast to single-skill tasks, long-horizon tasks play a crucial role in our daily life, e.g., a pouring task requires a proper concatenation of reaching, grasping and pouring subtasks. As an efficient solution for transferring human skills to robots, imitation learning has achieved great progress over the last two decades. However, when learning long-horizon visuomotor skills, imitation learning often demands a large amount of semantically segmented demonstrations. Moreover, the performance of imitation learning could be susceptible to external perturbation and visual occlusion. In this paper, we exploit dynamical movement primitives and meta-learning to provide a new framework for imitation learning, called Meta-Imitation Learning with Adaptive Dynamical Primitives (MiLa). MiLa allows for learning unsegmented long-horizon demonstrations and adapting to unseen tasks with a single demonstration. MiLa can also resist external disturbances and visual occlusion during task execution. Real-world robotic experiments demonstrate the superiority of MiLa, irrespective of visual occlusion and random perturbations on robots.

Keywords: Computer Vision for Automation, Deep Learning for Visual Perception, Logistics
Abstract: Trajectory prediction is an essential component of the perception stack in autonomous mobile robots (AMRs). AMRs operate in complex environments where their movements are influenced by various environment elements, such as racks and storage locations. Therefore, accurate and efficient trajectory prediction for intralogistics requires detailed environment modeling that goes beyond the lane-based context mainly used in road traffic methods. We propose the addition of a new environment context encoder module that can be seamlessly integrated into state-of-the-art autonomous driving systems. Our approach, tailored to the specific challenges of intralogistics, achieves highly accurate predictions using compact and efficient baseline networks.

Keywords: Computer Vision for Automation, SLAM
Abstract: Incrementally recovering real-sized 3D geometry from a pose-free RGB stream is a challenging task in 3D reconstruction, requiring minimal assumptions on input data. Existing methods can be broadly categorized into end-to-end and visual SLAM-based approaches, both of which either struggle with long sequences or depend on slow test-time optimization and depth sensors. To address this, we first integrate a depth estimator into an RGB-D SLAM system, but this approach is hindered by inaccurate geometric details in predicted depth. Through further investigation, we find that 3D Gaussian mapping can effectively solve this problem. Building on this, we propose an online 3D reconstruction method using 3D Gaussian-based SLAM, combined with a feed-forward recurrent prediction module to directly infer camera pose from optical flow. This approach replaces slow test-time optimization with fast network inference, significantly improving tracking speed. Additionally, we introduce a local graph rendering technique to enhance robustness in feedforward pose prediction. Experimental results on the Replica and TUM-RGBD datasets, along with a real-world deployment demonstration, show that our method achieves performance on par with the state-of-the-art SplaTAM, while reducing tracking time by more than 90%. Code is available at https://github.com/wangyr22/DepthGS

Keywords: Computer Vision for Automation, Computer Vision for Transportation, Vision-Based Navigation
Abstract: Mobile robots necessitate advanced natural language understanding capabilities to accurately identify locations and perform tasks such as package delivery. However, traditional visual place recognition (VPR) methods rely solely on single-view visual information and cannot interpret human language descriptions. To overcome this challenge, we bridge text and vision by proposing a multiview (360° views of the surroundings) text-vision registration approach called Text4VPR for place recognition task, which is the first method that exclusively utilizes textual descriptions to match a database of images. Text4VPR employs the frozen T5 language model to extract global textual embeddings. Additionally, it utilizes the Sinkhorn algorithm with temperature coefficient to assign local tokens to their respective clusters, thereby aggregating visual descriptors from images. During the training stage, Text4VPR emphasizes the alignment between individual text-image pairs for precise textual description. In the inference stage, Text4VPR uses the Cascaded Cross-Attention Cosine Alignment (CCCA) to address the internal mismatch between text and image groups. Subsequently, Text4VPR performs precisely place match based on the descriptions of text-image groups. On Street360Loc, the first text to image VPR dataset we created, Text4VPR builds a robust baseline, achieving a leading top-1 accuracy of 57% and a leading top-10 accuracy of 92% within a 5-meter radius on the test set, which indicates that localization from textual descriptions to images is not only feasible but also holds significant potential for further advancement, as shown in Figure 1. Our code is available at https://github.com/nuozimiaowu/Text4VPR.

Keywords: Computer Vision for Automation, RGB-D Perception, Vision-Based Navigation
Abstract: In recent years, 3D Gaussian Splatting (3D-GS)-based scene representation demonstrates significant potential in real-time rendering and training efficiency. However, most existing methods primarily focus on single-map reconstruction, while the registration and fusion of multiple 3D-GS sub-maps remain underexplored. Existing methods typically rely on manual intervention to select a reference sub-map as a template and use point cloud matching for registration. Moreover, hard-threshold filtering of 3D-GS primitives often degrades rendering quality after fusion. In this paper, we present a novel approach for automated 3D-GS sub-map alignment and fusion, eliminating the need for manual intervention while enhancing registration accuracy and fusion quality. First, we extract geometric skeletons across multiple scenes and leverage ellipsoid-aware convolution to capture 3D-GS attributes, facilitating robust scene registration. Second, we introduce a multi-factor Gaussian fusion strategy to mitigate the scene element loss caused by rigid thresholding. Experiments on the ScanNet-GSReg and our Coord datasets demonstrate the effectiveness of the proposed method in registration and fusion. For registration, it achieves a 41.9% reduction in RRE on complex scenes, ensuring more precise pose estimation. For fusion, it improves PSNR by 10.11 dB, highlighting superior structural preservation. These results confirm its ability to enhance scene alignment and reconstruction fidelity, ensuring more consistent and accurate 3D scene representation for robotic perception and autonomous navigation.

Keywords: Computer Vision for Automation, Computer Vision for Transportation, Recognition
Abstract: This study addresses the challenge of robust object detection in maritime environments, where dynamic conditions such as fog, brightness variations, and motion blur can degrade accuracy. We propose a novel framework, Joint Semantic Learning (JSL), which combines ocean scene segmentation and object detection to improve both performance and robustness. JSL incorporates the ocean scene segmentation module into the detection network during training and removes it during inference, ensuring no additional computational overhead. Through ocean scene segmentation, the feature extractor learns to understand the overall context of the image and extract detailed information about objects. Extensive experiments show that JSL, applied to various convolutional neural network-based detectors, achieves significant performance improvements on maritime datasets SMD and SeaShips. Notably, the proposed method shows substantial performance gains on the SMD-C and SeaShips-C datasets, which include adverse conditions, demonstrating the robustness of the proposed method. Furthermore, experiments comparing our method with existing state-of-the-art multi-task methods on the Cityscapes dataset validate its effectiveness in generalizing to urban environments. The efficient integration of spatial and semantic information of JSL ensures accurate and reliable object detection across diverse applications. Our code is available at: https://github.com/gist-ailab/JSL.

Keywords: Computer Vision for Automation, Visual Learning
Abstract: Semantic Scene Completion (SSC) aims to reconstruct the entire 3D scene in terms of both occupancy and semantics, serving as a fundamental task for autonomous driving and robotic systems. Camera-based methods have seen significant advancements due to their low cost and rich visual cues. However, previous approaches have predominantly focused on semantic recovery. This can lead to inaccurate occupancy predictions and, consequently, the failure of downstream tasks such as trajectory planning. To address this limitation, we propose a novel multi-frame matching framework, GeoScene, which reconstructs spatial structures through inter-frame geometric correlations of temporal images and subsequently infers scene semantic information. Specifically, we extract features from distinct frames in the depth dimension and derive depth features by constructing a cost volume. Following this, dot product and voxelization operations are applied between the extracted features and depth features to correct assignment errors. Furthermore, we introduce a surface normal-based regression loss to preserve fine-grained surface structures. Extensive experiments on the SemanticKITTI dataset demonstrate that GeoScene outperforms existing state-of-the-art methods.

Keywords: Computer Vision for Automation, Deep Learning for Visual Perception, AI-Based Methods
Abstract: LiDAR scene generation is critical for mitigating real-world LiDAR data collection costs and enhancing the robustness of downstream perception tasks in autonomous driving. However, existing methods commonly struggle to capture geometric realism and global topological consistency. Recent LiDAR Diffusion Models (LiDMs) predominantly embed LiDAR points into the latent space for improved generation efficiency, which limits their interpretable ability to model detailed geometric structures and preserve global topological consistency. To address these challenges, we propose TopoLiDM, a novel framework that integrates graph neural networks (GNNs) with diffusion models under topological regularization for high-fidelity LiDAR generation. Our approach first trains a topological-preserving VAE to extract latent graph representations by graph construction and multiple graph convolutional layers. Then we freeze the VAE and generate novel latent topological graphs through the latent diffusion models. We also introduce 0-dimensional persistent homology (PH) constraints, ensuring the generated LiDAR scenes adhere to real-world global topological structures. Extensive experiments on the KITTI-360 dataset demonstrate TopoLiDM’s superiority over state-of-the-art methods, achieving improvements of 22.6% lower Fréchet Range Image Distance (FRID) and 9.2% lower Minimum Matching Distance (MMD). Notably, our model also enables fast generation speed with an average inference time of 1.68 samples/s, showcasing its scalability for real-world applications. We will release the related codes at https://github.com/IRMVLab/TopoLiDM.

Keywords: Computer Vision for Automation, Object Detection, Segmentation and Categorization, Deep Learning Methods
Abstract: The rapid advancement of drone technology has led to the widespread application of micro-object detection in Unmanned Aerial Vehicle (UAV) systems. However, with the constraint of real-time computation, critical challenges remain in addressing extreme scale variations, low-resolution signatures and dense occlusions. For object detection task, although YOLO-based detectors outperform transformer models in efficiency-accuracy balance, their limited capacity for global context modeling and feature discriminability in complex aerial environments hinders optimal performance. To overcome these limitations, we introduce UAV-MaLO, a novel framework that incorporates state space modeling principles into YOLO's architecture. By introducing the abilities of long-range dependency modeling and adaptive spatial-frequency fusion, the proposed approach dynamically optimizes receptive fields while suppressing background interference, achieving robust micro-object localization in cluttered scenarios. Furthermore, the parallelized attention mechanism and the hierarchical feature refinement further ensure real-time processing capabilities without compromising detection precision, establishing a new paradigm for UAV deployment. Our experimental results on the VisDrone-2019-DET dataset reveal a significant improvement in various variants of average precision (AP), indicating the extraordinary performance of our UAV-MaLO.

Keywords: Prosthetics and Exoskeletons, Task and Motion Planning, Body Balancing
Abstract: Powered exoskeletons offer a promising solution for individuals who require assistance with daily activities. In addition to walking on flat terrains, they are evolving to handle advanced scenarios such as stairs, allowing for a wider range of activities. To facilitate this advancement, an effective and safe tool is required for the planning and generation of gait trajectories. This study proposes a novel online

trajectory generation method for an exoskeleton to aid in stair ascent.

Initially, a finite state machine model is designed to assist the user in shifting their center of gravity. Subsequently, a B´ezier curve-based path interpolation approach is introduced to generate the path between each state. Finally, a time scaling and path reparameterization method is employed to avoid the singularity problem. Two kinds of numerical simulations and real exoskeleton experiments demonstrate that the proposed method can effectively and safely generate the trajectories to assist users in ascending stairs. Additionally, the effectiveness of the exoskeleton’s assistance is verified through a comparison of electromyography (EMG) signals from seven muscles. The results show that there was a reduction in muscle activation ranging from 22% to 81% for the different muscles analyzed.

Keywords: Prosthetics and Exoskeletons, Physically Assistive Devices, Intention Recognition
Abstract: Gait phase detection is crucial to realize personalized assistive functions of lower limb exoskeletons. A common method in gait phase estimation is the adaptive oscillator, which performs well in periodic gaits. However, these types of methods fail in gait phase estimation under aperiodic gait cycles. Although some modified methods have been proposed for gait phase estimation under multiple terrains, they usually require dataset training, and the estimation accuracy is highly dependent on the collected dataset. To realize accurate and stable recognition of gait phase, a novel gait phase estimation method is proposed to estimate the gait phase without dataset training. This method, by incorporating discrete wavelet transform (DWT) with adaptive oscillators (AOs), can identify non-periodic mutations online and reset the oscillator at an appropriate time to avoid the divergence phenomenon when the adaptive oscillator is subjected to mutation signals. In this proposed method, the hip angle is measured by an inertial measurement unit (IMU) sensor and the measured data is then processed using discrete wavelet transform to detect the maximum hip flexion angle (MFA) and non-periodic mutations. The gait phase is finally estimated by a modified adaptive oscillator. Ground walking tests with a variety of speeds by six subjects were conducted under different walking conditions and the results show that the new method performs well in gait phase estimation under multiple terrains. The method is demonstrated on a hip exoskeleton in walking assistance.

Keywords: Prosthetics and Exoskeletons, Human-Centered Robotics, Reinforcement Learning
Abstract: Soft exoskeleton robots (exosuits) have exhibited promising potentials in walking assistance with comfortable wearing experience. In this paper, a twisted string actuator (TSA) is developed and equipped with the exosuit to provide powerful driving force and variable assistance intensity for hemiplegic patients, which provides human-domain and robotdomain training modes for subjects with different movement capabilities. Since the human-exosuit coupling dynamics is difficult to be modeled due to the soft structure of the exosuit and incomplete knowledge of the wearer’s performance, accurate control and efficient assistance cannot be guaranteed in current exosuits. By taking advantage of the motion characteristic of hemiplegic patients, a mirror adaptive impedance control is proposed, where the robotic actuation is modulated based on the motion and physiological reference of the healthy limb (HL) as well as the performance of the impaired limb (IL). A linear quadratic regulation (LQR) is formulated to minimize the bilateral trajectory tracking errors and human effort, and the adaptation between the human-domain and robot-domain modes can be realized. A reinforcement learning (RL) algorithm is designed to solve the given LQR problem to optimize the impedance parameters with little information of the human or robot model. The proposed robotic system is validated through experiments to perform its effectiveness and superiority.

Keywords: Prosthetics and Exoskeletons, Model Learning for Control, Multi-Modal Perception for HRI
Abstract: Exoskeleton technology holds significant promise within the human-centric paradigm of Industry 5.0 for mitigating work-related musculoskeletal disorders (WMSDs). However, existing systems often struggle with mismatched assistive torque and inefficient human-machine collaboration under dynamic loading conditions, largely due to insufficient motion intent recognition accuracy. This study proposes a dual-model-based multimodal fusion control strategy that integrates a bidirectional LSTM neural network (Bi-LSTM) with a transformer-based multi-task learning model (MTL) to enable real-time torque compensation and accurate prediction of dynamic load mass under varying conditions. The team developed a lightweight elbow joint exoskeleton prototype, leveraging multi-modal information to enhance assistive torque prediction accuracy. Experimental results show an 83.7% reduction in agonist muscle activation under a 3.5 kg load compared to conditions without the exoskeleton, underscoring its potential for industrial material handling scenarios.

Keywords: Prosthetics and Exoskeletons, Rehabilitation Robotics, Intention Recognition
Abstract: Recognizing and identifying human locomotion is a critical step to ensuring fluent control of wearable robots, such as transtibial prostheses. In particular, classifying the intended locomotion mode and estimating the gait phase are key. In this work, a novel, interpretable, and computationally efficient algorithm is presented for simultaneously predicting locomotion mode and gait phase. Using able-bodied (AB) and transtibial prosthesis (PR) data, seven locomotion modes are tested including slow, medium, and fast level walking (0.6, 0.8, and 1.0 m/s), ramp ascent/descent (5 degrees), and stair ascent/descent (20 cm height). Overall classification accuracy was 99.1% and 99.3% for the AB and PR conditions, respectively. The average gait phase error across all data was less than 4%. Exploiting the structure of the data, computational efficiency reached 2.91μs per time step. The time complexity of this algorithm scales as O(N·M) with the number of locomotion modes M and samples per gait cycle N. This efficiency and high accuracy could accommodate a much larger set of locomotion modes (∼700 on Open-Source Leg Prosthesis) to handle the wide range of activities pursued by individuals during daily living.

Keywords: Prosthetics and Exoskeletons, Human-Aware Motion Planning
Abstract: Achieving natural locomotion across diverse environments with prosthetic limbs remains a significant challenge for amputees. Intelligent prosthetics leverage motion planning techniques using phase variables to emulate natural gait aligned with human movement intentions. However, traditional phase variable-based planning, which utilizes geometric human motion models, often lacks robustness when encountering external disturbances. Additionally, models derived from human walking data can only approximate a limited set of discrete tasks, hindering the construction of a comprehensive model. In this study, we present an advanced prosthetic motion planning approach that integrates Dynamic Motion Primitives (DMPs) to ensure robust performance across multiple tasks. We demonstrate that DMPs with human-in-the-loop effectively simulate human joint movement trajectories under various task conditions. Furthermore, we introduce a novel Multi-Task Dynamic Motion Primitives with Singular Value Decomposition (DMPs-SVD) method, which incorporates multiple feature trajectory learning. This approach constructs a coherent task model using a limited dataset of typical human walking patterns, enabling joint motion planning across diverse task scenarios. Experimental results validate the viability and efficacy of the proposed human-in-loop DMPs and DMPs-SVD techniques in prosthetic applications.

Keywords: Prosthetics and Exoskeletons, AI-Based Methods, Marine Robotics
Abstract: By accurately recognizing the wearer’s motion, the underwater exoskeleton enables more efficient human-machine collaboration and provides enhanced assistance in complex and dynamic underwater environments. In this study, we propose a soft underwater exosuit motion mode recognizer based on a long short-term memory network and convolutional neural networks, referred to as LSTM-CNN. This model is designed to perform two tasks: motion mode classification and state transition label recognition. First, the LSTM network extracts features from the time-series data, followed by further feature extraction and classification using the convolutional and fully connected networks. The recognition of motion modes relies on three IMU sensors placed on the left and right legs and the back of the torso of the soft underwater exosuit. On the dataset containing four classes, including non-assist, breaststroke, flutter kick, and underwater walking, LSTM-CNN achieved an overall accuracy of 99.943±0.006% in motion mode classification and 92.101±0.054% in state transition label recognition. The experimental results indicate that the LSTM-CNN achieves better accuracy and performs optimally across various evaluation metrics compared to the other methods.

Keywords: Prosthetics and Exoskeletons, Human and Humanoid Motion Analysis and Synthesis
Abstract: The virtual pivot point (VPP), a theoretical convergence point of ground reaction forces during gait, has gained attention for its potential to uncover underlying locomotor control strategies. Here, we present the first investigation of VPP in individuals with unilateral above-knee amputation, using a publicly available dataset of 18 participants. Subjects were categorized into K2 (walking speeds 0.4–0.8 m/s) and K3 (0.6–1.4 m/s) functional levels. Our findings show that both groups demonstrate high sagittal-plane VPP quality, comparable to that of healthy individuals, with R2 > 95%, indicating a strong relationship between VPP formation and sagittal plane dynamics. Conversely, in the frontal plane, VPP analysis reveals greater variability and lower quality, indicating the absence of a well-defined pivot during gait. Notably, frontal-plane VPP quality deteriorates with increasing walking speed, particularly in K3 ambulators. While this speed-dependency is observed in healthy individuals as well, the rate of decline is significantly steeper in amputees. Additionally, spatial analysis of VPP positions reveals a consistent elevation of the amputated leg’s VPP compared to the intact leg. These findings emphasize the importance of frontal plane dynamics in amputee gait and suggest improvements in prosthetic design to enhance control and promote more symmetrical, natural gait.

Keywords: Multi-Robot Systems, Intelligent Transportation Systems, Human-Robot Teaming
Abstract: Urban intersections with diverse vehicle types, from small cars to large semi-trailers, pose significant challenges for traffic control. This study explores how robot vehicles (RVs) can enhance heterogeneous traffic flow, particularly at unsignalized intersections where traditional methods fail during power outages. Using reinforcement learning (RL) and real-world data, we simulate mixed traffic at complex intersections with RV penetration rates ranging from 10% to 90%. Results show that average waiting times drop by up to 86% and 91% compared to signalized and unsignalized intersections, respectively. We observe a "rarity advantage," where less frequent vehicles benefit the most (up to 87%). Although CO2 emissions and fuel consumption increase with RV penetration, they remain well below those of traditional signalized traffic. Decreased space headways also indicate more efficient road usage. These findings highlight RVs' potential to improve traffic efficiency and reduce environmental impact in complex, heterogeneous settings.

Keywords: Intelligent Transportation Systems, Datasets for Human Motion, Simulation and Animation
Abstract: Extreme weather and infrastructure vulnerabilities pose significant challenges to urban mobility, particularly at intersections where signals become inoperative. To address this growing concern, we introduce Beacon, a naturalistic driving dataset capturing traffic dynamics during blackouts at two major intersections in Memphis, TN, USA. The dataset provides detailed traffic movements, including timesteps, origin, and destination lanes for each vehicle over four hours of peak periods. We analyze traffic demand, vehicle trajectories, and density across different scenarios, demonstrating high-fidelity reconstruction under unsignalized, signalized, and mixed traffic conditions. We find that integrating robot vehicles (RVs) into traffic flow can substantially reduce intersection delays, with wait time improvements of up to 82.6%. However, this enhanced traffic efficiency comes with varying environmental impacts, as decreased vehicle idling may lead to higher overall CO2 emissions. To the best of our knowledge, Beacon is the first publicly available traffic dataset for naturalistic driving behaviors during blackouts at intersections.

Keywords: Intelligent Transportation Systems, Motion and Path Planning, Human-Aware Motion Planning
Abstract: Safety assurance using all perceptual information to predict the motion of dynamic agents is critical in urban environments and remains an open challenge. For Autonomous Vehicles (AV) operating around vulnerable road users, the risk assessment strategy often needs to address stochastic uncertainties in the multiple possible trajectories (or multimodal motion) of the surrounding traffic agents. However, this increases the complexity of the navigation problem using the existing planners. To address this issue, this paper presents a multi-level optimization strategy that combines sampling-based and direct optimization methods for decision-making and control with improved safety and trajectory smoothness. In the primary stage, a sampling-based optimization framework systematically identifies safe candidate trajectories by employing the Fusion of stochastic Predictive Inter-Distance Profile (F-sPIDP). F-sPIDP encapsulates the multimodal dynamics of traffic agents and explicitly computes the uncertainties in their estimated or tracked states. From the set of trajectories, a reference optimal trajectory and its F-sPIDP setpoints are selected, adhering to stringent safety constraints and motion smoothness. Subsequently, a secondary local control optimization refines the optimal trajectory to ensure compliance with the AV’s kinematic and dynamic constraints while accounting for the quantified uncertainty within the F-sPIDP framework. The performance of the proposed method was assessed through simulations and statistical analyses, evaluating its robustness to diverse levels of uncertainty.

Keywords: Intelligent Transportation Systems, Reinforcement Learning, Autonomous Vehicle Navigation
Abstract: Driving decision-making in mixed traffic, characterized by high-dynamic interactions and stochastic behaviors of human-driven vehicles, poses significant challenges for autonomous driving systems. To address these issues, we propose a novel Transformer-based Spatial Temporal Fusion (TSTF) module integrated with an auxiliary contrastive learning task within a multi-agent reinforcement learning (MARL) framework. The TSTF module captures interaction-aware behaviors and long-term temporal dependencies that tackle mixed cooperative driving scenarios, while the auxiliary contrastive learning task refines feature representations to enhance exploration efficiency and decision stability. Experimental evaluations on the MetaDrive platform demonstrate that the proposed approach outperforms baseline algorithms in safety, adaptability and robustness to dynamic traffic scenarios. The results highlight the effectiveness of the TSTF module in enabling robust and context-aware collaborative driving behaviors, offering a scalable solution for real-world mixed traffic. This work advances MARL by addressing key challenges in interaction modeling and driving decision-making under uncertainty, with significant implications for the development of intelligent transportation systems.

Keywords: Intelligent Transportation Systems, Deep Learning for Visual Perception, Semantic Scene Understanding
Abstract: Trajectory forecasting in urban environments is a critical task that needs to be addressed for the safety of autonomous vehicles, particularly in urban road intersection scenarios, where agents exhibit diverse behaviors mainly due to complex interactions between agents and the environment and the diversity of paths available to the agents. Current state-of-the-art methods do not perform well in urban road intersection scenarios. To address this issue, the proposed novel framework CandidateGraph-Net (CG-Net), improves trajectory prediction in urban road intersection scenarios by encoding the available candidate centerlines at the current location of the target agent. The proposed interaction encoder in CG-Net is inspired by human behavior. It is modeled utilizing a bipartite graph attention network to predict the trajectory of the target agent. It estimates the trajectory in the same way as humans anticipate the trajectory of other vehicles and pedestrians in dynamic environments. The agent embeddings in the interaction encoder at each time step pay attention to nearby agents and surrounding scene elements simultaneously. This enables the model to learn how to prioritize interactions between nearby agents and the environment map. Further, CG-Net's performance is evaluated using the Argoverse 2 motion forecasting dataset. The results demonstrate its effectiveness in urban road intersection scenarios, with an overall improvement in key metrics such as minFDE and minADE compared to baseline methods. These improvements highlight CG-Net’s ability to perform better motion forecasting in urban road intersection scenarios.

Keywords: Intelligent Transportation Systems, AI-Based Methods, Behavior-Based Systems
Abstract: Automated parking is a critical feature of Advanced Driver Assistance Systems (ADAS), where accurate trajectory prediction is essential to bridge perception and planning modules. Despite its significance, research in this domain remains relatively limited, with most existing studies concentrating on single-modal trajectory prediction of vehicles. In this work, we propose ParkDiffusion, a novel approach that predicts the trajectories of both vehicles and pedestrians in automated parking scenarios. ParkDiffusion employs diffusion models to capture the inherent uncertainty and multi-modality of future trajectories, incorporating several key innovations. First, we propose a dual map encoder that processes soft semantic cues and hard geometric constraints using a two-step cross-attention mechanism. Second, we introduce an adaptive agent type embedding module, which dynamically conditions the prediction process on the distinct characteristics of vehicles and pedestrians. Third, to ensure kinematic feasibility, our model outputs control signals that are subsequently used within a kinematic framework to generate physically feasible trajectories. We evaluate ParkDiffusion on the Dragon Lake Parking (DLP) dataset and the Intersections Drone (inD) dataset. Our work establishes a new baseline for heterogeneous trajectory prediction in parking scenarios, outperforming existing methods by a considerable margin.

Keywords: Intelligent Transportation Systems, Deep Learning Methods
Abstract: Deep learning-based trajectory prediction models for autonomous driving often struggle with generalization to out-of-distribution (OOD) scenarios, sometimes performing worse than simple rule-based models. To address this limitation, we propose a novel framework, Adaptive Prediction Ensemble (APE), which integrates deep learning and rule-based prediction experts. A learned routing function, trained concurrently with the deep learning model, dynamically selects the most reliable prediction based on the input scenario. Our experiments on large-scale datasets, including Waymo Open Motion Dataset (WOMD) and Argoverse, demonstrate improvement in zero-shot generalization across datasets. We show that our method outperforms individual prediction models and other variants, particularly in long-horizon prediction and scenarios with a high proportion of OOD data. This work highlights the potential of hybrid approaches for robust and generalizable motion prediction in autonomous driving.

Keywords: AI-Based Methods, Acceptability and Trust, Intelligent Transportation Systems
Abstract: Accurate trajectory prediction is vital for various applications, including autonomous vehicles. However, the complexity and limited transparency of many prediction algorithms often result in black-box models, making it challenging to understand their limitations and anticipate potential failures. This further raises potential risks for systems based on these prediction models. This study introduces the performance monitor for the black-box trajectory prediction model (PMBP) to address this challenge. The PMBP estimates the performance of black-box trajectory prediction models online, enabling informed decision-making. The study explores various methods' applicability to the PMBP, including anomaly detection, machine learning, deep learning, and ensemble, with specific monitors designed for each technique to provide online output representing prediction performance. Comprehensive experiments validate the PMBP's effectiveness, comparing different monitoring methods. Results show that the PMBP effectively achieves promising monitoring performance, particularly excelling in deep learning-based monitoring. It achieves improvement scores of 0.81 and 0.79 for average prediction error and final prediction error monitoring, respectively, outperforming previous white-box and gray-box methods. Furthermore, the PMBP's applicability is validated on different datasets and prediction models, while ablation studies confirm the effectiveness of the proposed mechanism. Hybrid prediction and autonomous driving planning experiments further show the PMBP's value from an application perspective.

Keywords: Cellular and Modular Robots, Swarm Robotics, Multi-Robot Systems
Abstract: This paper presents a framework for producing robotic fabrics using square lattice formations of interlinked Kilobot modules. The framework supports: (i) fabrics of arbitrary size and shape; (ii) different types of deformable links, namely springs and rods; (iii) easy plug-and-play reconfigurability. Two decentralized straight motion controllers are tested with robotic fabrics comprising up to 81 physical modules: an open-loop controller and a controller from the literature that responds to deformations within the fabric. For spring-based robotic fabrics, the deformation-correcting controller performs best overall, whereas for rod-based robotic fabrics, it is outperformed by the open-loop controller. A decentralized turning motion controller is formally derived and examined for either type of fabric, revealing the ability of the robotic fabrics to move along a curved trajectory using open-loop control. Finally, robotic fabrics are shown to perform basic object manipulation tasks. Robotic fabrics that deploy themselves based on distributed, embodied intelligence could pave the way for novel applications, from patching broken pipes to medical uses within the human body.

Keywords: Multi-Robot Systems, Path Planning for Multiple Mobile Robots or Agents, Task and Motion Planning
Abstract: This letter presents a topology-accelerated and robust pursuit framework for environments with obstacles considering state measurement uncertainty. Our framework consists of three primary components: the selection of virtual target points using topological heuristic method, the computation of safe pursuit regions based on Voronoi cell (VC) and the solution of an adaptive robust path controller based on Control Barrier Function (CBF). Topological heuristics broadly capture the topological structure of the environment and provide guidance for the selection of target points for each pursuer. Then the chance constrained obstacle-aware Voronoi cell (CCOVC) for each pursuer is constructed by calculating separation hyperplane and buffer terms. Finally, we formulate chance CBF and chance Control Lyapunov Function (CLF) constraints, using convex approximation to determine their upper bounds. We then find the adaptive robust path controller by solving a Quadratic Constraint Quadratic Programming (QCQP). Benchmark simulation and experimental results demonstrate the efficiency and robustness of the proposed framework.

Keywords: Multi-Robot Systems, Reinforcement Learning, Cooperating Robots
Abstract: In this letter, we introduce ArenaSim, a novel simulation platform designed for realistic and efficient self-play learning in multi-robot cooperative-competitive games. Compared to previous simulation platforms designed for the same task, we achieve fine-grained simulation of the robots with rotatable gimbals and roller-independent mecanum wheels in ArenaSim and validate its fidelity through real-world experiments. To inspire further exploration of this simulation platform, we design a hierarchical structure to address the cooperative-competitive game. The hierarchical structure is composed of a high-level strategy that generates macro actions such as move and shoot, and a low-level controller that translates these macro actions into precise motion control. Furthermore, we evaluate several self-play algorithms in ArenaSim and present a benchmark. The experiments show that the multi-robot cooperative-competitive game is still challenging for self-play learning. We hope that ArenaSim can further inspire research on self-play learning and multi-robot cooperative-competitive games.

Keywords: Multi-Robot Systems, Reinforcement Learning
Abstract: To ensure system responsiveness, some computeintensive tasks are usually offloaded to cloud or edge computing devices. In environments where connection to external computing facilities is unavailable, computation offloading among members within an autonomous multi-robot system (AMRS) becomes a solution. The challenge lies in how to maximize the use of other members’ idle resources without disrupting their local computation tasks. Therefore, this study proposes HRL-AMRS, a hierarchical deep reinforcement learning framework designed to distribute computational loads and reduce the processing time of computational tasks within an AMRS. In this framework, the high-level must consider the impact of data loading scales determined by low-level under varying computational device states on the actual processing times. In addition, the low-level employs Long Short-Term Memory (LSTM) networks to enhance the understanding of time-series states of computing devices. Experimental results show that, across various task sizes and numbers of robots, the framework reduces processing times by an average of 4.32% compared to baseline methods.

Keywords: Multi-Robot Systems, Task Planning, Distributed Robot Systems
Abstract: This paper develops two distributed algorithms to solve multi-robot task assignment problems (MTAP). We first describe MTAP as an integer linear programming (ILP) problem and then reformulate it as a relaxed convex optimization problem. Based on the saddle-point dynamics, we propose two distributed optimization algorithms using optimistic gradient decent ascent (OGDA) and extra-gradient (EG) methods, which achieve exact convergence to an optimal solution of the relaxed problem. In most cases, such an solution reflects the optimality of the original ILP problems. For some special ILP problems, we provide a perturbation-based distributed method to avoid the inconsistency phenomenon, such that an optimal solution to any ILP problem is obtained. Compared with some decentralized algorithms requiring a central robot that communicates with the other robots, our developed algorithms are fully distributed, in which each robot only communicates with the nearest neighbors for an arbitrary connected graph. We evaluate the developed algorithms in terms of computation, communication, and data storage complexities, and compare them with some typical algorithms. It is shown that the developed algorithms have low computational and communication complexities. We also verify the effectiveness of our algorithms via numerical examples.

Keywords: Multi-Robot Systems
Abstract: Long-term monitoring of numerous dynamic targets can be tedious for a human operator and infeasible for a single robot, e.g., to monitor wild flocks, detect intruders, search and rescue. Fleets of autonomous robots can be effective by acting collaboratively and concurrently. However, the online coordination is challenging due to the unknown behaviors of the targets and the limited perception of each robot. Existing work often deploys all robots available without minimizing the fleet size, or neglects the constraints on their resources such as battery and memory. This work proposes an online coordination scheme called LOMORO for collaborative target monitoring, path routing and resource charging. It includes three core components: (I) the modeling of multi-robot task assignment problem under the constraints on resources and monitoring intervals; (II) the resource-aware task coordination algorithm iterates between the high-level assignment of dynamic targets and the low-level multi-objective routing via the Martin's algorithm; (III) the online adaptation algorithm in case of unpredictable target behaviors and robot failures. It ensures the explicitly upper-bounded monitoring intervals for all targets and the lower-bounded resource levels for all robots, while minimizing the average number of active robots. The proposed methods are validated extensively via large-scale simulations against several baselines, under different road networks, robot velocities, charging rates and monitoring intervals.

Keywords: Multi-Robot Systems
Abstract: Euler-Lagrange systems are often used to describe many practical plants with strong coupling nonlinearities, such as roboticmanipulators, autonomous surface vehicles, and wearable exoskeletons. Since a single Euler-Lagrange system has limited capabilities, it is imperative to construct multiple Euler-Lagrange systems (MELSs) to collaborate on complex operational missions. However, most of the existing controllers for MELSs only obtain asymptotic stabilization, which cannot guarantee the system states fast stabilization. Furthermore, the sampling and transmission of data at high frequencies within the MELSs can lead to network congestion, thus affecting the stability of the system. Hence, a distributed neural fixed-time consensus controller with an event-triggered mechanism is presented to not only improve the leader-following consensus speed under a directed communication graph, but also save the communication resources of the MELSs. Ultimately, the effectiveness of the proposed controller is verified through simulations and experimental results about a multiple manipulator system.

Keywords: Multi-Robot Systems
Abstract: The Parasitic capacitance of magnetic components, i.e., inductors and transformers, are crucial, because it dominates the high-frequency impedance, causes voltage / current spikes, and arouses electromagnetic interference (EMI) issues. The existing methods characterize capacitance from all five parts (turn-to-turn, layer-to-layer, winding-to-magnetic core, winding-to-electrostatic screen, and interwinding). However, the calculated results are always less than the measured capacitance, especially for the inductors with few winding turns. This article discovers another mechanism of the parasitic capacitance caused by dB/dt. It points out that a time-varying magnetic field also generates an electric field, which leads to parasitic capacitance, and proves that this capacitance dominates the first resonant frequency (denoted by fR) of the inductors with few winding turns. The factors related to the capacitance and the fR are analyzed. The proposed theoretical analysis is validated by numerous simulations and experimental results. All the simulations are included in the multimedia folder.

Keywords: Modeling, Control, and Learning for Soft Robots, Biologically-Inspired Robots, Soft Robot Applications
Abstract: Maneuverability in soft bio-inspired underwater robots, particularly for following complex trajectories, remains an unsolved challenge. In this work, we present a control approach based on a PD controller integrated with real-time camera feedback, enabling continuous and reliable free-swimming control. The system was able to maintain precise waypoint tracking for 60 minutes and more, with a minimum turning radius of 27 cm, demonstrating the robot's high maneuverability relative to its body size. Our contributions include the development of a robust camera-based tracking system, the tuning of a PD controller to enhance trajectory following, and the exploration of the limits of maneuverability in soft swimming robots. This work paves the way for future integration with onboard sensing systems to improve state estimation in soft swimmers and reduce reliance on external camera systems.

Keywords: Modeling, Control, and Learning for Soft Robots, Sensor Fusion, AI-Based Methods
Abstract: Soft, elastic platforms may pose an intricate challenge towards sensor fusion as forces acting on the structure render extrinsic transformations variable over time. The present paper tackles this problem by introducing an elastic deformation model and embedding it into a sensor fusion scheme. The core of our method is given by a neural representation mapping temporal deformation sequences onto mass-normalized restoring forces. By using continuous time trajectory models as well as Newton’s second law, the sensor fusion problem becomes solvable by enforcing the consistency between second-order trajectory differentials and network outputs. The approach is validated on a loosely-coupled, real-world fusion scenario: an elastically connected, non-overlapping stereo camera system. As demonstrated, our approach permits relative camera alignment, absolute scale recovery, as well as inertial alignment from individual visual odometry results.

Keywords: Modeling, Control, and Learning for Soft Robots, Soft Robot Applications
Abstract: Soft manipulators (SMs) have shown great potential for interactive tasks in confined environments. However, avoiding obstacles of SMs may conflict with the manipulator’s configuration, the planned trajectory for tracking control, and the target position for grasping. To coordinate configuration planning, tracking control, and target grasping in obstacle avoidance, this study proposes a hierarchical configuration planning framework with three levels: behavior planning, configuration planning, and shape/position control. At the behavior planning level, a Discrete Event System (DES)-based planner is designed to orchestrate mode transitions among obstacle avoidance, tracking control, and target grasping. The configuration planning level adopts the Bézier curve to model the SM backbone curve and constructs a repulsive potential field to quantify obstacle effects on the entire manipulator configuration. Under the constraints of grasping distance and material physical limit, the control points of the Bézier curve corresponding to the optimal configuration that minimizes the repulsive potential energy are computed. Experiments demonstrate the effectiveness of the proposed framework in achieving collision-free configuration planning for object grasping and placement in confined operational scenarios.

Keywords: Modeling, Control, and Learning for Soft Robots, Soft Robot Applications, Soft Robot Materials and Design
Abstract: In this paper, we employ multiple IMUs into a triple-section continuum manipulator to precisely capture the attitude data of each section's end disk. Leveraging the sensory and mechanical hardware system, we construct a sophisticated coordinate transformation scheme to accurately identify the detailed configuration states of the manipulator. Additionally, we introduce a numerical optimization strategy to develop a unified forward and inverse kinematic modeling framework, ensuring both iterative efficiency and accuracy. Through the IMUs' real-time attitude feedback, we implement a closed-loop controller, enhancing the manipulator's operational robustness and agility. In our experimental evaluations, we assess the convergence performance of both forward and inverse kinematics within a simulated environment and validate the precision of these kinematic models through real-time experiments on an actual continuum manipulator. Moreover, we evaluate the performance of the proposed controller by examining its accuracy during the manipulator's continuous motions and analyzing its response characteristics. In contrast to previous research on continuum robots in the literature, we pioneer a fully integrated kinematic control architecture that is successfully implemented on a physical continuum robotic system.

Keywords: Modeling, Control, and Learning for Soft Robots, Soft Robot Applications, AI-Based Methods
Abstract: Soft robotic systems heavily depend on accurate sensor data for perception and control; however, this data is often corrupted by missing observations, due to partial sensor coverage, communication failures, or occlusions and noisy measurements stemming from hardware imperfections, environmental disturbances, and the intrinsic compliance of soft materials. Such corruption can obscure critical state information, causing unreliable modeling of soft robotics and degrading control accuracy. To address these challenges, we propose a Contrastive Dual-Latent Autoencoder (CDLAE) that jointly handles missing and noisy data in a single end-to-end framework. Our approach leverages an attention based autoencoder architecture with dual latent pathways, where one focuses on capturing the underlying clean signals while the other isolates noise-related components. A contrastive loss encourages strong separation between these pathways, enhancing the model’s ability to filter noise while reconstructing missing values. Additionally, the autoencoder is trained jointly with a downstream predictive network, ensuring that signal imputation is optimized with respect to the ultimate control task. Experimental evaluations on a pneumatic soft robot platform and multiple public time-series datasets demonstrate that CDLAE consistently outperforms existing methods in handling corrupted data, offering robust, high-fidelity reconstructions that significantly improve soft robot perception and control in real-world conditions.

Keywords: Modeling, Control, and Learning for Soft Robots, Bioinspired Robot Learning, Biologically-Inspired Robots
Abstract: Soft-bodied crawling animals exhibit efficient and adaptive behaviors resulting from the synergy between morphological computation (e.g., a flexible soft body and anisotropic skin) and neural computation (e.g., neural control with plasticity and short-term memory (STM)). However, applying these principles to soft crawling robots remains challenging. To address this, our study proposes an adaptive neural control system that incorporates online learning and STM to generate adaptive behaviors in soft crawling robots. This control system was implemented in a robot with a flexible soft body, anisotropic abdominal denticles or skin, and embodied laser and flex sensors. The robot demonstrated a multilevel adaptation to various perturbations. Perturbations, such as rough terrain, can be managed through passive (body) adaptation via micro-deformation of the denticles and macro-deformation of the body. Larger perturbations, including being lifted or pressed, navigating confined spaces, and traversing slopes, are handled by active (neural control) adaptation. The robot can learn new behaviors, such as navigating confined spaces, and store sensory information to maintain the learned behavior robustly, even in the temporary absence of sensory feedback. In addition, it can estimate its state through sensory feedback prediction, detect abnormal states through prediction errors, and adapt its behavior to address these errors.

Keywords: Modeling, Control, and Learning for Soft Robots, Biologically-Inspired Robots, Soft Sensors and Actuators
Abstract: With the progress in ocean exploration, bionic soft robotic fish have garnered significant attention, with their key feature being the actuation mechanism made from soft materials. However, the complex properties of these materials pose challenges in modeling and control. In this letter, we design and fabricate a Fish-like Bionic Soft Actuation Mechanism (FBSAM) and aim to achieve its tail oscillation control. First, we construct an experimental platform to collect data on FBSAM's motion characteristics, revealing complex nonlinear hysteresis influenced by varying liquid environments. Next, we develop a phenomenological model for FBSAM based on the Hammerstein architecture and identify its parameters via nonlinear least squares algorithm. Subsequently, we propose an integral sliding mode hybrid control strategy, introducing an inverse hysteresis compensator to address hysteresis issue and using the neural network to estimate uncertain disturbances caused by liquid environments. Finally, experimental results demonstrate that the designed FBSAM can oscillate in water like a real fish, and the proposed control strategy adapts to various external environments, maintaining excellent performance even in dynamic flow conditions, showcasing its effectiveness and superiority.

Keywords: Modeling, Control, and Learning for Soft Robots, Biologically-Inspired Robots, Hydraulic/Pneumatic Actuators, Marine Robotics
Abstract: This paper explores a hydraulically powered double-joint soft robotic fish called HyperTuna and a set of locomotion optimization methods. HyperTuna has an innovative, highly efficient actuation structure that includes a four-cylinder piston pump and a double-joint soft actuator with self-sensing. We conducted deformation analysis on the actuator and established a finite element model to predict its performance. A closed-loop strategy combining a central pattern generator controller and a proportional–integral–derivative controller was developed to control the swimming posture accurately. Next, a dynamic model for the robotic fish was established considering the soft actuator, and the model parameters were identified via data-driven methods. Then, a particle swarm optimization algorithm was adopted to optimize the control parameters and improve the locomotion performance. Experimental results showed that the maximum speed increased by 3.6% and the cost of transport (COT) decreased by up to 13.9% at 0.4 m/s after optimization. The proposed robotic fish achieved a maximum speed of 1.12 BL/s and a minimum COT of 12.1 J/(kg·m), which are outstanding relative to those of similar soft robotic fish. Lastly, HyperTuna completed turning and diving–floating movements and long-distance continuous swimming in open water, which confirmed its potential for practical application.

Keywords: Multifingered Hands, In-Hand Manipulation, Reinforcement Learning
Abstract: Manipulating articulated tools, such as tweezers or scissors, has rarely been explored in previous research. Unlike rigid tools, articulated tools change their shape dynamically, creating unique challenges for dexterous robotic hands. In this work, we present a hierarchical, goal-conditioned reinforcement learning (GCRL) framework to improve the manipulation capabilities of anthropomorphic robotic hands using articulated tools. Our framework comprises two policy layers: (1) a low-level policy that enables the dexterous hand to manipulate the tool into various configurations for objects of different sizes, and (2) a high-level policy that defines the tool's goal state and controls the robotic arm for object-picking tasks. We employ an encoder, trained on synthetic pointclouds, to estimate the tool's affordance states—specifically, how different tool configurations (e.g., tweezer opening angles) enable grasping of objects of varying sizes—from input point clouds, thereby enabling precise tool manipulation. We also utilize a privilege-informed heuristic policy to generate replay buffer, improving the training efficiency of the high-level policy. We validate our approach through real-world experiments, showing that the robot can effectively manipulate a tweezer-like tool to grasp objects of diverse shapes and sizes with a 70.8% success rate. This study highlights the potential of RL to advance dexterous robotic manipulation of articulated tools.

Keywords: Multifingered Hands, Model Learning for Control, Manipulation Planning
Abstract: This paper investigates a framework (CATCH-FORM-3D) for the precise contact force control and surface deformation regulation in viscoelastic material manipulation. A partial differential equation (PDE) is proposed to model the spatiotemporal stress-strain dynamics, integrating 3D Kelvin–Voigt (stiffness-damping) and Maxwell (diffusion) effects to capture the material’s viscoelastic behavior. Key mechanical parameters (stiffness, damping, diffusion coefficients) are estimated in real time via a PDE-driven observer. This observer fuses visual-tactile sensor data and experimentally validated forces to generate rich regressor signals. Then, an inner-outer loop control structure is built up. In the outer loop, the reference deformation is updated by a novel admittance control law, i.e., a proportional-derivative (PD) feedback law with contact force measurements, ensuring that the system responds adaptively to external interactions. In the inner loop, a reaction-diffusion PDE for the deformation tracking error is formulated and then exponentially stabilized by conforming the contact surface to analytical geometric configurations (i.e., defining Dirichlet boundary conditions). This dual-loop architecture enables the effective deformation regulation in dynamic contact environments. Experiments using a PaXini robotic hand demonstrate sub-millimeter deformation accuracy and stable force tracking (pm 5 % deviation). The framework advances compliant robotic interactions in applications like industrial assembly, polymer shaping, surgical treatment, and household service.

Keywords: Multifingered Hands, Grasping, Mechanism Design
Abstract: Humanoid robotic hands need to be versatile and capable of providing environmental information in order to serve as a platform for intelligent grasp control. To facilitate the design process of such hands, we present the KIT Robotic Hands. They have been designed to meet diverse application requirements through their scalability in size, actuation, sensorization and computing resources. The hands integrate a multi-modal sensor system, in-hand embedded processing capabilities, an adaptive underactuated mechanism and a continuously controllable thumb rotation to enhance dexterity. The flexibility of the design is demonstrated through two application-specific hand implementations: one is the ARMAR-7 hand, which has human hand dimensions for grasping daily objects in household tasks, the other is the ARMAR-DE hand, a larger hand designed for grasping bigger objects in decontamination tasks. We describe the design and mechatronics of the hands as well as an evaluation of the grasp success and image segmentation based on an in-hand integrated camera and onboard processing of visual data.

Keywords: Multifingered Hands, Telerobotics and Teleoperation, Dexterous Manipulation
Abstract: This paper addresses the scarcity of affordable, fully-actuated five-fingered hands for dexterous teleoperation, which is crucial for collecting large-scale real-robot data within the "Learning from Demonstrations'" paradigm. We introduce the prototype version of the RAPID Hand, the first low-cost, 20-degree-of-actuation (DoA) dexterous hand that integrates a novel anthropomorphic actuation and transmission scheme with an optimized motor layout and structural design to enhance dexterity. Specifically, the RAPID Hand features a universal phalangeal transmission scheme for the non-thumb fingers and an omnidirectional thumb actuation mechanism. Prioritizing affordability, the hand employs 3D-printed parts combined with custom gears for easier replacement and repair. We assess the RAPID Hand's performance through quantitative metrics and qualitative testing in a dexterous teleoperation system, which is evaluated on three challenging tasks: multi-finger retrieval, ladle handling, and human-like piano playing. The results indicate that the RAPID Hand’s fully actuated 20-DoF design holds significant promise for dexterous teleoperation. The project will be open-sourced.

Keywords: Multifingered Hands, Perception for Grasping and Manipulation, Grasping
Abstract: The deep learning models has significantly advanced dexterous manipulation techniques for multi-fingered hand grasping. However, the contact information-guided grasping in cluttered environments remains largely underexplored. To address this gap, we have developed a method for generating multi-fingered hand grasp samples in cluttered settings through contact semantic map. We introduce a contact semantic conditional variational autoencoder network (CoSe-CVAE) for creating comprehensive contact semantic map from object point cloud. We utilize grasp detection method to estimate hand grasp poses from the contact semantic map. Finally, an unified grasp evaluation model PointNetGPD++ is designed to assess grasp quality and collision probability, substantially improving the reliability of identifying optimal grasps in cluttered scenarios. Our grasp generation method has demonstrated remarkable success, outperforming state-of-the-art (SOTA) methods by at least 4.7%, with 81.0% average grasping success rate in real-world single-object grasping using a known hand, and by at least 9.0% when using an unknown hand. Moreover, in cluttered scenes, our method attains a 76.7% success rate, outperforming the SOTA method by 6.3%. We also proposed the multi-modal multi-fingered grasping dataset generation method. Our multi-fingered hand grasping dataset outperforms previous datasets in scene diversity, modality diversity. More details and supplementary materials can be found at https://sites.google.com/view/contact-dexnet.

Keywords: Multifingered Hands, Dexterous Manipulation, Tendon/Wire Mechanism
Abstract: General-purpose robots should possess human-like dexterity and agility to perform tasks with the same versatility as us. A human-like form factor further enables the use of vast datasets of human-hand interactions. However, the primary bottleneck in dexterous manipulation lies not only in software but arguably even more in hardware. Robotic hands that approach human capabilities are often prohibitively expensive, bulky, or require enterprise-level maintenance, limiting their accessibility for broader research and practical applications. What if the research community could get started with reliable dexterous hands within a day? We present the open-source ORCA hand, a reliable and anthropomorphic 17-DoF tendon-driven robotic hand with integrated tactile sensors, fully assembled in less than eight hours and built for a material cost below 2,000 CHF. We showcase ORCA's key design features such as popping joints, auto-calibration, and tensioning systems that significantly reduce complexity while increasing reliability, accuracy, and robustness. We benchmark the ORCA hand across a variety of tasks, ranging from teleoperation and imitation learning to zero-shot sim-to-real reinforcement learning. Furthermore, we demonstrate its durability, withstanding more than 10,000 continuous operation cycles---equivalent to approximately 20 hours---without hardware failure, the only constraint being the duration of the experiment itself. Video is here: youtu.be/kUbPSYMmOds. All design files, source code, and documentation are available at orca.ethz.ch.

Keywords: Manipulation Planning, Motion and Path Planning, Grasping
Abstract: Robot pick and place systems have traditionally decoupled grasp, placement, and motion planning to build sequential optimization pipelines with an assumption that the individual components will be able to work together. However, this separation introduces sub-optimality, as grasp choices may limit, or even prohibit, feasible motions for a robot to reach the target placement pose, particularly in cluttered environments with narrow passages. To this end, we propose a forest-based planning framework to simultaneously find grasp configurations and feasible robot motions that explicitly satisfy downstream placement configurations paired with the selected grasps. Our proposed framework leverages a bidirectional sampling-based approach to build a start forest, rooted at the feasible grasp regions, and a goal forest, rooted at the feasible placement regions, to facilitate the search through randomly explored motions that connect valid pairs of grasp and placement trees. We demonstrate that the framework’s inherent parallelism enables superlinear speedup, making it scalable for applications for redundant robot arms, e.g., 7 DoF, to work efficiently in highly cluttered environments. Extensive experiments in simulation demonstrate the robustness and efficiency of the proposed framework in comparison with multiple baselines under diverse scenarios.

Keywords: Manipulation Planning, Mobile Manipulation, Autonomous Agents
Abstract: In this paper, we present SeGMan, a hybrid motion planning framework that integrates sampling-based and optimization-based techniques with a guided forward search to address complex, constrained sequential manipulation challenges, such as pick-and-place puzzles. SeGMan incorporates an adaptive subgoal selection method that adjusts the granularity of subgoals, enhancing overall efficiency. Furthermore, proposed generalizable heuristics guide the forward search in a more targeted manner. Extensive evaluations in maze-like tasks populated with numerous objects and obstacles demonstrate that SeGMan is capable of generating not only consistent and computationally efficient manipulation plans but also outperform state-of-the-art approaches.

Keywords: Force and Tactile Sensing, Recognition, Cyborgs
Abstract: 传统的磁性触觉传感器经常受到影响 外部磁场干扰。为了克服这个问题 问题，本工作提出并设计了一种双模态软 具有减轻磁性的能力的磁性皮肤 场干扰。灵感来自于感官机制 人体皮肤，设计的磁性皮肤可以捕捉 双模态信息同时，即磁性和 跨时空域的强制触觉信息。一个 卷积神经网络-卷积神经 构建网络多层感知器（CNN-CNN-MLP） 融合双模态触觉信息。为了 减少外部磁场干扰的影响， 一种新型动态加权系数层（DWCL）是 提出，它动态地为每个 基于实时输入特征的模态。 具体来说，通过分析 接触前传感和量化过程中的模态 目标物体的磁场强度，DWCL 自动调整融合比率以优先考虑 在不同干扰下具有更高可靠性的模态 条件。大量实验结果表明，DWCL 在干扰方面表现出显着改善 与传统融合策略相比的阻力。

Keywords: Force and Tactile Sensing, Touch in HRI, Haptics and Haptic Interfaces
Abstract: 以提供自然和身临其境的互动 人机背景下的远程作体验 协作互动，这项工作创新 构建自然的人机交互系统 基于超声波触觉反馈的远程作。 具体来说，它可以准确地捕捉操作员的手 在远程机器人上移动并复制这些动作 具有低延迟和高保真度。它利用 超声波相控阵实现非接触式触觉 反馈。我们提出了一种动态超声阵列声学 一种基于交互特征的字段定制方法 信息图像。该方法可以动态调整 根据操作员的手进行声场 特点，真实聚焦多个目标点 时间，并将它们投射到操作员的指尖上， 从而提供力可控的非接触式触觉 反馈给操作员。运算符集成到 我们系统的反馈回路，控制着系统 通过多模态反馈，形成高质量 人机交互封闭控制系统。最后， 系统的性能在两个经典机器人中得到验证 任务：块拾取和放置和螺母拧紧。这 实验结果表明，该系统表现出 出色的准确性和灵巧性，可以高效地 高精度完成任务，同时提供出色的 操作员的互动体验。

Keywords: Force and Tactile Sensing, Soft Sensors and Actuators, Sensor Networks
Abstract: Human-like embodied tactile perception is crucial for the next-generation intelligent robotics. Achieving large-area, full-body soft coverage with high sensitivity and rapid response, akin to human skin, remains a formidable challenge due to critical bottlenecks in encoding efficiency and wiring complexity in existing flexible tactile sensors, thus significantly hinder the scalability and real-time performance required for human skin-level tactile perception. Herein, we present a paradigm-shifting architecture employing code division multiple access- inspired orthogonal digital encoding to overcome these challenges. Our decentralized encoding strategy transforms conventional serial signal transmission by enabling parallel superposition of energy-orthogonal base codes from distributed sensing nodes, drastically reducing wiring requirements and increasing data throughput. We implemented and validated this strategy with off-the-shelf 16-node sensing array to reconstruct the pressure distribution, achieving a temporal resolution of 12.8 ms using only a single transmission wire. Crucially, the architecture can maintain sub-20ms latency across orders-of-magnitude variations in node number (to thousands of nodes). By fundamentally redefining signal encoding paradigms in soft electronics, this work opens new frontiers in developing scalable embodied intelligent systems with human-like sensory capabilities.

Keywords: Force and Tactile Sensing
Abstract: Vision and touch are two fundamental sensory modalities for robots, offering complementary information that enhances perception and manipulation tasks. Previous research has attempted to jointly learn visual-tactile representations to extract more meaningful information. However, these approaches often rely on direct combination, such as feature addition and concatenation, for modality fusion, which tend to result in poor feature integration. In this paper, we propose ConViTac, a visual-tactile representation learning network designed to enhance the alignment of features during fusion using contrastive representations. Our key contribution is a Contrastive Embedding Conditioning (CEC) mechanism that leverages a contrastive encoder pretrained through self-supervised contrastive learning to project visual and tactile inputs into unified latent embeddings. These embeddings are used to couple visual-tactile feature fusion through cross-modal attention, aiming at aligning the unified representations and enhancing performance on downstream tasks. We conduct extensive experiments to demonstrate the superiority of ConViTac in real world over current state-of-the-art methods and the effectiveness of our proposed CEC mechanism, which improves accuracy by up to 12.0% in material classification and grasping prediction tasks.

Keywords: Force and Tactile Sensing, Deep Learning Methods, Representation Learning
Abstract: Contact-rich manipulation remains a major challenge in robotics. Optical tactile sensors like GelSight Mini offer a low-cost solution for contact sensing by capturing soft-body deformations of the silicone gel. However, accurately inferring shear and normal force distributions from these gel deformations has yet to be fully addressed. In this work, we propose a machine learning approach using a U-net architecture to predict force distributions directly from the sensor's raw images. Our model, trained on force distributions inferred from Finite Element Analysis (FEA), demonstrates promising accuracy in predicting normal and shear force distributions for the commercially available GelSight Mini sensor. It also shows potential for generalization across indenters, sensors of the same type, and for enabling real-time application. The codebase, dataset and models are open-sourced and available at https://feats-ai.github.io.

Keywords: Force and Tactile Sensing
Abstract: Magnetic-based tactile sensors (MBTS) combine the advantages of compact design and high-frequency operation but suffer from limited spatial resolution due to their sparse taxel arrays. This paper proposes SuperMag, a tactile shape reconstruction method that addresses this limitation by leveraging high-resolution vision-based tactile sensor (VBTS) data to supervise MBTS super-resolution. Co-designed, open-source VBTS and MBTS with identical contact modules enable synchronized data collection of high-resolution shapes and magnetic signals via a symmetric calibration setup. We frame tactile shape reconstruction as a conditional generative problem, employing a conditional variational auto-encoder to infer high-resolution shapes from low-resolution MBTS inputs. The MBTS achieves a sampling frequency of 125 Hz, whereas the shape reconstruction sustains an inference time within 2.5 ms. This cross-modality synergy advances tactile perception of the MBTS, potentially unlocking its new capabilities in high-precision robotic tasks.

Keywords: Force and Tactile Sensing, Neurorobotics
Abstract: Neuromorphic sensors are a promising technology in artificial touch due to their low latency and low computational and power requirements, particularly when paired with spiking neural networks (SNNs). Here, we explore the ability of these systems to adapt to and generalize across varying sources of uncertainty in tactile tasks. We choose Braille reading as an application task and collect event-based data for 27 braille characters with a neuromorphic tactile sensor (NeuroTac) under varying conditions of tapping speed, center position and indentation depth using a 6-DOF robot arm. We initially analyze the effect of spatial location and speed on classification performance with spiking convolutional neural networks (SCNNs). We then show that SCNNs are able to generalize across each dimension. The final general SCNN model reaches 95.33% accuracy with uncertainty in all 4 dimensions. This research demonstrates the noise degradation performance of SCNNs in a tactile task, and outlines the potential of a single SCNN to generalize across several dimensions of uncertainty.

Keywords: Virtual Reality and Interfaces, Social HRI, Human-Robot Collaboration
Abstract: Translating human intent into robot commands is crucial for the future of service robots in an aging society. Existing Human-Robot Interaction (HRI) systems often rely on hand gestures or verbal commands, which can be restrictive and less practical for the elderly, making it challenging for them to remember complex word syntax or sign language. This paper introduces a multi-modal interaction framework that combines voice and deictic posture information to create a more natural HRI system. The visual cues are first processed by the object detection model to gain a global understanding of the environment, and then bounding boxes are estimated based on depth information. Using a large language model (LLM) and voice-to-text commands coupled with temporally aligned selected bounding boxes, sets of robot action sequences are generated. The action sequence is further constrained to key control syntax to prevent potential LLM hallucination issues. The system is evaluated on real-world tasks with varying levels of complexity using a Universal Robots UR3e 6-DoF robot arm. Compared to gesture control, utilizing parallel multi-modal sequences for robot control demonstrated a time-saving of up

Keywords: Tendon/Wire Mechanism, Model Learning for Control
Abstract: Cable-Driven Parallel Mechanisms (CDPMs) are subject to collision-free constraints from birth to the present. The collision-free constraints confine the workspace of CDPMs. To expand the workspace, scholars suggested allowing cables to wrap on rigid bodies (e.g., the end-effector) in recent years, opening new perspectives for CDPMs. However, the modeling and control of a CDPM with cables wrapped on general rigid bodies remain challenging. To this end, this study investigates the necessary conditions for a path of a cable of a CDPM wrapped on a smooth and frictionless rigid body. The solvability of the necessary conditions and the kinematics of the CDPM is explored then. It is shown that the kinematics of CDPMs with cables wrapped on rigid bodies, except for certain simple rigid bodies such as cylinders and spheres, is usually analytically unsolvable. To address the analytically unsolvable kinematics, the study develops a data-driven kinematic modeling and control strategy. The study applies the strategy to control the orientation of spatial rotational CDPM prototypes with wrapped cables and compares the strategy to a classical Jacobian-based kinematic control strategy. Experimental results indicate that for a CDPM allowing cables to wrap on a cylinder, the data-driven kinematic modeling and control strategy outperforms the Jacobian-based kinematic control strategy. For a CDPM allowing cables to wrap on a deformed cylinder, the data-driven kinematic modeling and control strategy can effectively control the CDPM.

Keywords: Tendon/Wire Mechanism, Medical Robots and Systems, Flexible Robotics
Abstract: The tendon-sheath mechanism (TSM) has significantly advanced both robotic systems and minimally invasive surgery (MIS) by enabling flexible and precise movement through narrow and tortuous paths. However, the inherent flexibility of TSM introduces nonlinear behaviors

which depend on its geometrical shape and applied forces, making accurate control challenging. Furthermore, the shape dependency becomes

critical in endoscopic robots, where the geometrical shape varies and is not directly visible, limiting the applicability of existing distal sensorless compensation methods. To address the geometry identification problem of TSM, this paper proposes an approach that utilizes real-time visual input from an endoscopic camera for on-line calibration of the TSM's physical model. By introducing the concept of the 'Equivalent Circle,' complex shapes of TSMs are simplified, enabling the estimation of their equivalent geometry without direct observation or measurement. Simulation results validate the equivalent circle model, demonstrating minimal deadband percentage errors despite larger discrepancies in equivalent radii across varied configurations. On-line calibration experiments achieved a percent error of 1.38% (±2.92%) for accumulated curve angles and 2.32% (±3.08%) for equivalent radii, demonstrating the

method’s reliability in shape estimation across varying conditions. In prediction and feedforward experiments, leveraging the equivalent circle to compensate for deadband in arbitrarily shaped TSMs resulted in a maximum trajectory error of 0.25 mm and an RMSE of 0.09 mm. This approach advances distal sensorless control, improving the operational accuracy and feasibility of endoscopic surgical robots under varying geometrical and force conditio

Keywords: Tendon/Wire Mechanism, Biomimetics, Flexible Robotics
Abstract: The excellent swimming ability of fish provides a new solution for the design of underwater propellers. However, there is a lack of optimization schemes for the structure and performance of the continuum-driving fish-like propeller. This paper introduces a continuum-driving fish-like propeller capable of online adjustments to the ratio of its active part and passive part (RAP). Then, a hydrodynamic model of the designed platform is established based on the being gradually softened approach (BGSA) to analyze the active and passive bending of the caudal fin. Finally, the coupling effect of RAP and the passive part stiffness (PS) on the propulsion performance of the fish-like propeller is explored through experiments. Results indicate that matching RAP and PS with the kinematic characteristics is essential for optimal propulsion performance. With the increase of flapping frequency, RAP and PS need to be increased. This work provides valuable guidance and insights for the design and optimization of the continuum driving fish-like propellers.

Keywords: Underactuated Robots, Body Balancing, Motion Control
Abstract: The spatial underactuated robot, which consists of two links moving in 3D space, serves as a simplified model for human postural control. The robot is supported at a single point, with 2 degrees of freedom (DoF) underactuated at the ankle joint and 2-DoF actuated at the hip joint. This paper proposes an angular momentum-based feedback controller for the robot. The controller can locally stabilize the robot to a given equilibrium point and enable transitions between different equilibrium points. Firstly, this paper extends the angular momentum model from planar to 3D space, revealing that the robot's balance behavior is determined by eight key parameters closely related to the choice of state variables. This paper analyzes the impact of these parameters on the controllability of the angular momentum model. Secondly, this paper provides analytical expressions for the control gain parameters using the pole placement method. Finally, this paper identifies the equilibrium points where the controller may encounter singularities and demonstrates through simulations that the controller can effectively stabilize the robot to a given equilibrium point and enable transitions between different equilibrium points while maintaining balance.

Keywords: Wheeled Robots, Engineering for Robotic Systems, Optimization and Optimal Control
Abstract: The wheeled-legged multi-mode vehicle (WLMV) combines the merits of both wheel and quadruped platforms and is gradually becoming a major development for high-mobility transportation in complex scenarios. To enhance the motility of the WLMV in challenging terrains, this paper proposes a bi-level optimization closed-loop model reference adaptive vibration control method (BO-CMRAC). The parameters and control weights of the reference model are determined through a bi-level optimization approach, which includes offline optimization and online weights selection, to coordinate motility and vibration suppression capabilities across varying scenarios. The dynamic characteristics of the actuator are modeled and incorporated into the closed-loop reference model to further improve the robustness and performance of the controller. A WLMV experimental platform is established to validate the proposed vibration control strategy on different roads. The results indicate that the proposed BO-CMRAC can effectively suppress the vibration of the WLMV compared to current algorithms.

Keywords: Tendon/Wire Mechanism, Visual Servoing, Surgical Robotics: Steerable Catheters/Needles
Abstract: In this letter, a constrained visual predictive control strategy (C-VPC) is developed for a robotic flexible endoscope to precisely track target features in narrow environments while adhering to visibility and joint limit constraints. The visibility constraint, crucial for keeping the target feature within the camera's field of view, is explicitly designed using zeroing control barrier functions to uphold the forward invariance of a visible set. To automate the robotic endoscope, kinematic modeling for image-based visual servoing is conducted, resulting in a state-space model that facilitates the prediction of the future evolution of the endoscopic state. The C-VPC method calculates the optimal control input by optimizing the model-based predictions of the future state under visibility and joint limit constraints. Both simulation and experimental results demonstrate the effectiveness of the proposed method in achieving autonomous target tracking and addressing the visibility constraint simultaneously. The proposed method achieved a reduction of 12.3% in Mean Absolute Error (MAE) and 56.0% in variance (VA) compared to classic IBVS.

Keywords: Control Architectures and Programming, Nonholonomic Mechanisms and Systems, Foundations of Automation
Abstract: Automating the sewing process presents significant challenges due to the inherent softness of fabrics and the limited control capabilities of sewing systems. To realize sewing automation, we propose a time-scaling modeling and control architecture of the robotic sewing system. By using the time-scaling modeling, the nonholonomic kinematics of the sewing process of the industrial sewing machine is linearized precisely. Based on this model, a two-layer real-time control architecture is proposed. The upper layer controls the sewn seam line trajectory using the model-based feedback control implemented in the time-scaling domain, while the lower layer controls the manipulator and the sewing machine using geometric-based trajectory generation and coordinated motion control of the robot and the sewing system in the time domain. The experimental results demonstrate that the sewing trajectories exponentially converge to the desired trajectories without overshooting under different initial conditions and sewing speeds. Besides, the same sewing trajectories under different sewing speeds are obtained for a given stitch size. The sewing results show the good performance and application potential of the proposed robotic sewing system.

Keywords: Path Planning for Multiple Mobile Robots or Agents, Motion and Path Planning, Imitation Learning
Abstract: Multi-Agent Path Finding (MAPF), which focuses on finding collision-free paths for multiple robots, is crucial for applications ranging from aerial swarms to warehouse automation. Solving MAPF is NP-hard so learning-based approaches for MAPF have gained attention, particularly those leveraging deep neural networks. Nonetheless, despite the community's continued efforts, all learning-based MAPF planners still rely on decentralized planning due to variability in the number of agents and map sizes. We have developed the first centralized learning-based policy for MAPF problem called RAILGUN. RAILGUN is not an agent-based policy but a map-based policy. By leveraging a CNN-based architecture, RAILGUN can generalize across different maps and handle any number of agents. We collect trajectories from rule-based methods to train our model in a supervised way. In experiments, RAILGUN outperforms most baseline methods and demonstrates great zero-shot generalization capabilities on various tasks, maps and agent numbers that were not seen in the training dataset.

Keywords: Path Planning for Multiple Mobile Robots or Agents, Optimization and Optimal Control
Abstract: This paper considers a multi-robot trajectory planning problem with inter-robot connectivity maintenance for information gathering. Given an information map in the form of a distribution over the workspace, ergodic search plans trajectories, along which, the time spent in any region is proportional to the amount of information in that region, and can balance between exploration and exploitation. Existing ergodic search rarely considers the limited communication range among robots or connectivity maintenance, and this paper takes a step to fill this gap. Besides, multi-robot connectivity maintenance was studied a lot, including continual, periodic, intermittent connectivity, etc. Naively combining these methods with ergodic search may prevent the planner from finding high-quality ergodic trajectories or lead to poor connectivity among the robots. To handle the challenge, this paper adapts an intermittent connectivity maintenance strategy to the ergodic search framework, and develops a two-phase trajectory planning approach utilizing the augmented Lagrangian method. Our simulation and real drone experiments show that under the same connectivity maintenance requirement, our approach plans trajectories that are about 10 times better than the baselines in terms of the ergodic metric.

Keywords: Path Planning for Multiple Mobile Robots or Agents, Cellular and Modular Robots, Assembly
Abstract: Self-assembly planning for modular robots is critical for constructing functional structures, yet existing methods often suffer from inefficiency, poor scalability, or collision risks. This paper presents an innovative framework that formulates modular robot self-assembly as a time-varying online Multi-Agent Path Finding (MAPF) problem and resolves it through an enhanced Time-Expanded Network (TEN). Key modifications are introduced to handle the dynamic nature of the self-assembly process, including the varying number of agents and evolving target configurations. Simulations conducted with hexagonal modular robots demonstrate that the proposed algorithm significantly outperforms the benchmark A*-based approach in terms of both assembly efficiency and success rate across various target configurations. The proposed framework establishes a scalable planning framework for modular robot self-assembly, with future extensions toward real-world validation.