Keywords: Legged Robots, Optimization and Optimal Control, Machine Learning for Robot Control
Abstract: Due to high dimensionality and non-convexity, real-time optimal control using full-order dynamics models for legged robots is challenging. Therefore, Nonlinear Model Predictive Control (NMPC) approaches are often limited to reduced-order models. Sampling-based MPC has shown potential in nonconvex even discontinuous problems, but often yields suboptimal solutions with high variance, which limits its applications in high-dimensional locomotion. This work introduces DIAL-MPC (Diffusion-Inspired Annealing for Legged MPC), a sampling-based MPC framework with a novel diffusion-style annealing process. Such an annealing process is supported by the theoretical landscape analysis of Model Predictive Path Integral Control (MPPI) and the connection between MPPI and single-step diffusion. Algorithmically, DIAL-MPC iteratively refines solutions online and achieves both global coverage and local convergence. In quadrupedal torque-level control tasks, DIAL-MPC reduces the tracking error of standard MPPI by 13.4 times and outperforms reinforcement learning (RL) policies by 50% in challenging climbing tasks without any training. In particular, DIAL-MPC enables precise real-world quadrupedal jumping with payload. To the best of our knowledge, DIAL-MPC is the first training-free method that optimizes over full-order quadruped dynamics in real-time.

Keywords: Deep Learning in Grasping and Manipulation, Grasping, Dexterous Manipulation
Abstract: Dexterous grasping is a fundamental yet challenging skill in robotic manipulation, requiring precise interaction between robotic hands and objects. In this paper, we present D(R,O) Grasp, a novel framework that models the interaction between the robotic hand in its grasping pose and the object, enabling broad generalization across various robot hands and object geometries. Our model takes the robot hand's description and object point cloud as inputs and efficiently predicts kinematically valid and stable grasps, demonstrating strong adaptability to diverse robot embodiments and object geometries. Extensive experiments conducted in both simulated and real-world environments validate the effectiveness of our approach, with significant improvements in success rate, grasp diversity, and inference speed across multiple robotic hands. Our method achieves an average success rate of 87.53% in simulation in less than one second, tested across three different dexterous robotic hands. In real-world experiments using the LeapHand, the method also demonstrates an average success rate of 89%. D(R,O) Grasp provides a robust solution for dexterous grasping in complex and varied environments. The code, appendix, and videos are available on our project website at https://nus-lins-lab.github.io/drograspweb/.

Keywords: Aerial Systems: Mechanics and Control, Dynamics, Motion Control
Abstract: Terrestrial and aerial bimodal vehicles have gained significant interest due to their energy efficiency and versatile maneuverability across different domains. However, most existing passive-wheeled bimodal vehicles rely on attitude regulation to generate forward thrust, which inevitably results in energy waste on producing lifting force. In this work, we propose a novel passive-wheeled bimodal vehicle called TrofyBot that can rapidly change the thrust direction with a single servo motor and a transformable parallelogram linkage mechanism (TPLM). Cooperating with a bidirectional force generation module (BFGM) for motors to produce bidirectional thrust, the robot achieves flexible mobility as a differential driven rover on the ground. This design achieves 95.37% energy saving efficiency in terrestrial locomotion, allowing the robot continuously move on the ground for more than two hours in current setup. Furthermore, the design obviates the need for attitude regulation and therefore provides a stable sensor field of view (FoV). We model the bimodal dynamics for the system, analyze its differential flatness property, and design a controller based on hybrid model predictive control for trajectory tracking. A prototype is built and extensive experiments are conducted to verify the design and the proposed controller, which achieves high energy efficiency and seamless transition between modes.

Keywords: Nonholonomic Motion Planning, Flexible Robotics, Kinematics
Abstract: Recent advancements in soft actuators have enabled soft continuum swimming robots to achieve higher efficiency and more closely mimic the behaviors of real marine animals. However, optimizing the design and control of these soft continuum robots remains a significant challenge. In this paper, we present a practical framework for the co-optimization of the design and control of soft continuum robots, approached from a geometric locomotion analysis perspective. This framework is based on the principles of geometric mechanics, accounting for swimming at both low and high Reynolds numbers. By generalizing geometric principles to continuum bodies, we achieve efficient geometric variational co-optimization of designs and gaits across different power consumption metrics and swimming environments. The resulting optimal designs and gaits exhibit greater efficiencies at both low and high Reynolds numbers compared to three-link or serpenoid swimmers with the same degrees of freedom, approaching or even surpassing the efficiencies of infinitely flexible swimmers and those with higher degrees of freedom.

Keywords: Force and Tactile Sensing, Dexterous Manipulation, Grasping
Abstract: To robustly handle objects, robots must perceive mechanical interactions through touch with sufficient richness. New tactile sensors leverage miniature cameras to provide dense measurements of these interactions, allowing for the extraction of material properties and frictional information. Among the plethora of solutions, retrographic sensing is popular for its ability to finely resolve the shape of the object being touched. This sensor uses a reflective membrane, illuminated at a shallow angle by three RGB lights, which cast colored shadows. From the illumination pattern of the deformed membrane, both the normal deformation and fine surface details can be recovered. However, these retrographic sensors cannot detect the lateral displacement of the membrane and, therefore, overlook frictional information, which is crucial for grasping and manipulation. Embedding and tracking opaque markers has been a makeshift solution, but these markers occlude the membrane and are difficult to manufacture. In this paper, we introduce ShadowTac, a tactile sensor that combines retrographic illumination with non-intrusive markers created by colored shadows. We patterned the retrographic surface with a dense array of submillimeter dimples, which are small enough not to obstruct the view yet cast shadows large enough to be visible to the camera. ShadowTac captures a dense image of both the normal displacement field with fine details and a precise lateral displacement field by tracking the markers. Additionally, our sensor is easy to manufacture, as the dimple pattern can simply be molded. We evaluated the measurement reliability of ShadowTac and its effectiveness in estimating the incipient slip of arbitrary objects. The dense measurement of both the normal and shear deformation that the sensor captures makes it ideal for tracking dynamic interactions between robotic fingertips and manipulated objects.

Keywords: Deep Learning in Grasping and Manipulation, Perception for Grasping and Manipulation, Grasping
Abstract: Reliable 6D pose estimation is crucial for robotic tasks but presents significant challenges in environments with occlusion. Recent approaches tend to directly predict pose parameters of object with deep neural networks, lacking the modeling ability of non-adjacent and complex relationships of surface points in occluded scenarios. To solve this problem, we propose a novel occlusion-aware 6D pose estimation framework, which uses depth-guided graph neural network (GNN) to model potential relationships from RGBD input. Two semantic information, which are mask and binary code of object, are adaptively fused to extract 2D-3D correspondence related features in an effective manner. Both enhanced graph features and fused semantic information contribute to the performance improvement of pose estimation with occlusion. Extensive experiments indicate that our approach outperforms comparative methods by 1.2% and 1.9% on LMO and YCBV datasets (up to 30% for certain objects) and its validity is also verified under real-world pose estimation test.

Keywords: SLAM, RGB-D Perception
Abstract: This paper presents a simultaneous localization and mapping (SLAM) system to provide accurate pose estimation and dynamic scene reconstruction. Our approach proposes a Joint Point-Gaussian Splatting representation, which fully integrates the robustness of isotropic feature points in pose estimation and the flexibility of anisotropic 3D Gaussians in scene representation. This system does not need to suppress the anisotropic representation of Gaussian elements, which enables the mapping module to achieve finer scene representation with lower memory consumption. Additionally, in order to enhance the adaptability of the system in dynamic environments, we introduced a dynamic region recognition module and utilized 3D Gaussian Splatting and 4D Gaussian Splatting representations to represent static and dynamic regions respectively. Furthermore, we developed a local map management strategy for Gaussian Splatting mapping, effectively reducing the memory and computational resource usage in the mapping process. Experiments on public datasets demonstrate that our system achieves state-of-the-art tracking and mapping accuracy compared to existing baselines.

Keywords: Sensor Fusion, Localization, SLAM
Abstract: Conventional LiDAR-inertial odometry (LIO) or SLAM methods heavily rely on geometric features of environments, as LiDARs primarily provide range measurements instead of motion measurements. From now on, however, the situation changes thanks to the novel Frequency Modulated Continuous Wave (FMCW) LiDARs. FMCW LiDARs not only offer the point range with high resolution but also capture the instant point Doppler velocity through the Doppler effect. In the letter, we propose FMCW-LIO, a novel and robust LIO, leveraging intrinsic Doppler measurements from FMCW LiDARs. To correctly exploit Doppler velocities, a motion compensation method is designed, and a Doppler-aided observation model is applied for on-manifold state estimation. Then, dynamic points can be effectively removed by the Doppler criteria, deriving more consistent geometric observations. FMCW-LIO eventually achieves accurate state estimation and static mapping, even in structure-degenerated environments. Extensive experiments in diverse scenes are performed and FMCW-LIO outperforms other algorithms on both accuracy and robustness.

Keywords: SLAM, Optimization and Optimal Control, Field Robots
Abstract: Keyframes are LiDAR scans saved for future reference in Simultaneous Localization And Mapping (SLAM), but despite their central importance most algorithms leave choices of which scans to save and how to use them to wasteful heuristics. This work proposes two novel keyframe selection strategies for localization and map summarization, as well as a novel approach to submap generation which selects keyframes that best constrain localization. Our results show that online keyframe identification and submap generation reduce the number of saved keyframes and improve per scan computation time without compromising localization performance. We also present a map summarization feature for quickly capturing environments under strict map size constraints.

Keywords: SLAM, Range Sensing, Mapping
Abstract: Range-only Simultaneous Localisation and Mapping (RO-SLAM) is of interest in the robotics community due to its practical applications; for example, ultra-wideband (UWB) and Bluetooth Low Energy (BLE) localisation in terrestrial and aerial applications and acoustic beacon localisation in marine applications. In this work, we consider a mobile robot equipped with an inertial measurement unit (IMU) and a range sensor that measures distances to a collection of fixed landmarks. We derive an equivariant filter (EqF) for the RO-SLAM problem based on a symmetry Lie group that is compatible with the range measurements. The proposed filter does not require bootstrapping or initialisation of landmark positions, and demonstrates robustness to the no-prior situation. The filter is demonstrated on a real-world dataset, and it is shown to significantly outperform a state-of-the-art EKF alternative in terms of both accuracy and robustness.

Keywords: Optimization and Optimal Control, Localization, Robot Safety, Global Optimality
Abstract: In recent years, semidefinite relaxations of common optimization problems in robotics have attracted growing attention due to their ability to provide globally optimal solutions. In many cases, it was shown that specific handcrafted redundant constraints are required to obtain tight relaxations, and thus global optimality. These constraints are formulation-dependent and typically identified through a lengthy manual process. Instead, the present article suggests an automatic method to find a set of sufficient redundant constraints to obtain tightness, if they exist. We first propose an efficient feasibility check to determine if a given set of variables can lead to a tight formulation. Second, we show how to scale the method to problems of bigger size. At no point of the process do we have to find redundant constraints manually. We showcase the effectiveness of the approach, in simulation and on real datasets, for range-based localization and stereo-based pose estimation. We also reproduce semidefinite relaxations presented in recent literature and show that our automatic method always finds a smaller set of constraints sufficient for tightness than previously considered.

Keywords: SLAM, Autonomous Vehicle Navigation, Mapping
Abstract: Abstract—Lidar-based Simultaneous Localization and Mapping (SLAM) encounters a substantial challenge in the form of accumulating errors, which can adversely impact its reliability. Loop closing techniques have been extensively employed to counteract this issue. Nonetheless, the loop closing conundrum remains difficult to resolve, as point clouds often exhibit partial overlap due to disparities in scanning pose (viewpoints). This renders the conventional point cloud registration such as Iterative Closest Point (ICP) algorithm problematic. To overcome this challenge, this paper proposes a two-stage viewpoint-aware point cloud registration technique that assigns suitable weights to the correspondence pairs associating two point clouds from different viewpoints. The weights account for the visibility of points from their respective viewpoint as well as from the viewpoint of the counterpart point cloud, making the registration more relying on commonly visible points from the both viewpoints. Experimental results, utilizing the KITTI and Apollo-SouthBay dataset, indicate that the proposed technique delivers more precise and robust performance compared to the baseline techniques.

Keywords: Actuation and Joint Mechanisms, Force Control, Tendon/Wire Mechanism
Abstract: This study presents a comprehensive experimental analysis of Twisted String Actuators (TSA), focused on enhancing contraction modelling accuracy and establishing a baseline for TSA tension and impedance control efficacy. A novel TSA string radius function is introduced, computing effective radii for multi-strand bundles based on axial actuator tension. The proposed model was validated in physical experiments, resulting in a reduction of maximal errors between measured and simulated actuator contraction trajectories from up to 60% in established models to around 10% in our work. Additionally, the tension-dependent radius modification effectively reduced errors between the estimated and the measured bundle tension by an order of magnitude, marking an essential step towards TSA control independent of bundle tension measurements. TSA tension control was assessed based on four metrics: accuracy, precision, impact stability, and bandwidth, following ISO 9283:1998 standards. The quality of tension control was found to be dependent on bundle tension, twisting angle and strand quantity, whereas impact stability was maintained in all configurations. Joint impedance control with TSA was evaluated for perturbation stability and position control bandwidth, where the latter was enhanced with increasing joint stiffness. The presented analysis informs designers about the capabilities of TSAs in different configurations, and their respective suitability for desired applications.

Keywords: Tendon/Wire Mechanism, Mechanism Design, Actuation and Joint Mechanisms
Abstract: This paper presents a novel actuator system combining a twisted string actuator (TSA) with a winch mechanism. Relative to traditional hydraulic and pneumatic systems in robotics, TSAs are compact and lightweight but face limitations in stroke length and force-transmission ratios. Our integrated TSA-winch system overcomes these constraints by providing variable transmission ratios through dynamic adjustment. It increases actuator stroke by winching instead of overtwisting, and it improves force output by twisting. The design features a rotating turret that houses a winch, which is mounted on a bevel gear assembly driven by a through-hole drive shaft. Mathematical models are developed for the combined displacement and velocity control of this system. Experimental validation demonstrates the actuator's ability to achieve a wide range of transmission ratios and precise movement control. We present performance data on movement precision and generated forces, discussing the results in the context of existing literature. This research contributes to the development of more versatile and efficient actuation systems for advanced robotic applications and improved automation solutions.

Keywords: Tendon/Wire Mechanism, Actuation and Joint Mechanisms, Medical Robots and Systems
Abstract: Cable transmissions are commonly used in robotics for remote force transmission, offering a lightweight, compact, and efficient solution for transmitting high forces between input and output. However, cables in flexible compression housings (Bowden cables), exhibit high static friction, which increases exponentially with total bend angle. Alternatively, internally routed ball-bearing supported cable capstan transmissions are low friction, but complex and present challenges in routing multiple sets of cables. In this paper, we propose motion-decoupled cable transmission modules that address these challenges, occupying the middle ground, functioning as discrete-joint ball-bearing supported Bowden cables. Our rolling-plus-twist joint design decouples pairs of routed cables from changing significantly in tension, length, or friction during large angle motion of the linked transmission. Using sub-1 mm diameter high-strength synthetic cable, the transmission exhibits a maximum coupling motion of only 0.15 mm over the full range of motion of the cable-transmission mechanism, approximately 10% of pretension in combined hysteresis and friction, a transmission stiffness of 10 N/mm, weighing just 9 g per rolling joint and 5 g per twist joint. Two applications are demonstrated: cable routing alongside a robot arm for, say, gripper remote actuation, and remote needle advancement for an MRI-safe needle biopsy robot.

Keywords: Actuation and Joint Mechanisms
Abstract: This study examines a novel setup of a micropositioning trajectory manipulator in Xθ, energized by a reluctance actuator (RA) and two accompanying moving magnet actuators (MMA). The design is characterized by a C-core RA, which features asymmetrical air gaps between the mover and the stator elements when under angular θ rotation. When the stator coil is energized, a magnetic flux induces a force in the mover. Two MMAs can add force and torque dynamics to the system via solenoid and permanent magnet (PM) pairs to offer additional corrective actions. Facilitating control of a translational x and rotational θ two-degree-of-freedom (2DOF) actuation system. Flexure hinges aid in the retraction force of the mover element and provide needed stiffness to the system without frictional effects. This was modeled analytically and optimized to achieve outlined performance objectives. The system was validated experimentally through triangle, and sinusoidal trajectories in open loop control. The most relevant application is scanning mirror systems where specific targeted rotational and translational trajectories can benefit light beam positioning. This system allows both translation and rotation specifications of a selected trajectory to be realized in one actuation unit, opening up more design possibilities for controlling precision positioning systems.

Keywords: Machine Learning for Robot Control, Actuation and Joint Mechanisms, Legged Robots
Abstract: This paper presents a novel approach through the design and implementation of Cycloidal Quasi-Direct Drive actuators for legged robotics. The cycloidal gear mechanism, with its inherent high torque density and mechanical robustness, offers significant advantages over conventional designs. By integrating cycloidal gears into the Quasi-Direct Drive framework, we aim to enhance the performance of legged robots, particularly in tasks demanding high torque and dynamic loads, while still keeping them lightweight. Additionally, we develop a torque estimation framework for the actuator using an Actuator Network, which effectively reduces the sim-to-real gap introduced by the cycloidal drive’s complex dynamics. This integration is crucial for capturing the complex dynamics of a cycloidal drive, which contributes to improved learning efficiency, agility, and adaptability for reinforcement learning.

Keywords: Mechanism Design, Compliant Joint/Mechanism, Robotic Wrist, Grasping
Abstract: We have developed a two-degree-of-freedom robotic wrist with variable stiffness capability, designed for situations where collisions between the end-effector and the environment are inevitable. To enhance environmental adaptability and prevent physical damage, the wrist can operate in a low-stiffness mode. However, the flexibility of this mode might negatively impact stable and precise manipulation. To address this, we proposed a robotic wrist that switches between a passive low-stiffness mode for environmental adaptation and an active high-stiffness mode for precise manipulation. Initially, we developed a functional prototype that could manually switch between these modes, demonstrating the wrist's passive low-stiffness and active high-stiffness states. This prototype was designed as a lightweight, flat-type modular device, incorporating a sheet-type flexure as the motion guide and embedding all essential components, including actuators, sensors, and a control unit, into the wrist module. Based on the functional prototype, we developed an improved version to enhance durability and functionality. The resulting wrist module incorporates a three-axis F/T sensor and an impedance control system to control the stiffness. It measures 55 mm in height, weighs 200 g, and offers a 232.4-fold active stiffness variation.

Keywords: Computer Vision for Automation, Computer Vision for Manufacturing, Recognition
Abstract: Active stereo vision (ASV) computes the parallax and depth information from the coded structured light patterns. Thus, it could overcome the difficulties of measuring objects without textures and colors. However, decoding of the structured light patterns at locations of color crosstalk, specular reflection and occlusion remains challenging. In this paper, we propose a natural-neighbor-interpolant-based pattern modeling method to decode the structured light point pattern robustly. The robustness is achieved in the sense of hundred percent point segmentation completeness. Due to the hundred percent completeness, the points in the corresponding blocks are matched directly according to their indexes. Experimental results verified the effectiveness of the proposed method.

Keywords: Computer Vision for Automation
Abstract: Video object detection (VOD) of motile cells (e.g., bacteria and sperm) under microscopy is challenging due to motion blur, sporadic out-of-focus, and pose variations. Compared with VOD in generic scenes, the lower contrast and smaller color space of microscopy imaging further introduce feature overlap between the foreground objects and the background objects (e.g., impurity cells and contaminants). Transformer-based methods have achieved great success in the VOD of generic scenes by utilizing object queries to model the inner-frame objects and the inter-frame objects. However, the appearance overlap problem in microscopy video frames significantly compromises the inter-frame query aggregation by introducing background features into the object query. To tackle this challenge, this paper reports a static-dynamic query-based VOD network that treats object queries of the current video frame and reference video frames differently. Specifically, a two-stage framework is implemented that first generates high-quality object queries of reference frames with a static Transformer decoder pre-trained on a still image dataset. The network is then trained on a per-frame annotated dataset using a dynamic Transformer decoder to model the object queries of the current frame. A Reference Query Relation Module is further proposed to enhance the reference queries for more effective aggregation with the current query. Experiments on clinically collected biopsied sperm datasets validated the effectiveness of the proposed method.

Keywords: Space Robotics and Automation, Mining Robotics, Learning from Demonstration
Abstract: To support sustainable infrastructure on the Moon, NASA is developing the In-Situ Resource Utilization (ISRU) Pilot Excavator (IPEx) to extract and transport lunar regolith for processing and construction. During its mission, IPEx will execute various driving patterns, primarily cycling between excavation and unloading sites, with additional maneuvers such as circular traverses around the lander and raster scans for environmental mapping. In this work, dynamic movement primitives (DMPs) are used to represent these patterns. We augment the DMPs with a vision-based real-time obstacle avoidance system to navigate surface hazards, such as rocks, encountered during traversal. Our approach is evaluated in a high-fidelity simulation replicating the challenging environment of the lunar south pole to demonstrate IPEx’s ability to adapt to surface hazards while fulfilling its operational tasks.

Keywords: Deep Learning for Visual Perception, Aerial Systems: Perception and Autonomy, Recognition
Abstract: Fire poses significant threats to life and property, necessitating efficient inspection and accurate identification. Although aerial computer vision algorithms hold great promise, the computational limitations of onboard platforms prevent existing algorithms from meeting high standards of accuracy and real-time performance. To address this challenge, we propose an lightweight aerial fire detection model, LAFNET. This model incorporates the EffiDarknetLight backbone, optimized for lightweight design, integrates specially designed LG block components within the LG PAN neck, resulting in a model Params of only 1.3M. Experimental results demonstrate that our method attains a good trade-off between lightweight design and detection accuracy. Compared to the smallest standard YOLO series' model YOLOv5n, LAFNET improves MAP by 2.1%, while reducing Params and FLOPs by 27.8% and 29.3%, the inference speed on Nvidia Orin Nano edge computing side improves 24.8%. These experiments indicate that LAFNET offers a highly efficient solution for aerial fire detection, combining speed and accuracy.

Keywords: Omnidirectional Vision, Computer Vision for Transportation, Intelligent Transportation Systems
Abstract: In recent years, with the rapid development of Advanced Driver Assistance Systems (ADAS), the demand for the precise and efficient surround view stitching system has significantly increased. Traditional stitching methods perform well in small single-unit vehicles with stable camera poses. However, the stitching quality sharply degrades when applied to large tractor-trailers due to the continuous pose changes caused by the non-rigid connection between the tractor and trailer. In detail, first, the extended length of tractor-trailers results in low overlap between cameras, making feature extraction and matching challenging. Additionally, the stitched images often appear irregular, detracting from visual quality. Besides, even if static stitching looks natural, it causes jitter in dynamic scenarios due to random feature extraction. In this paper, we propose an unsupervised deep stitching method for tractortrailer surround view system. We introduce a feature extraction module for tractor-trailer scenarios (FMT) to enhance feature extraction in low-overlap situations. Besides, we design a spatiotemporally consistent control point constraint strategy (STCC) to achieve spatial shape preservation and temporal smoothing effects, resulting in visually consistent and stable stitched sequences. Experimental results from both public and real dataset show that our method efficiently completes tractortrailer surround view stitching, producing well-aligned and natural panoramic images compared to previous methods.

Keywords: Computer Vision for Medical Robotics, Surgical Robotics: Laparoscopy, Visual Learning
Abstract: Despite advancements in robotic systems and surgical data science, ensuring safe execution in robot-assisted minimally invasive surgery (RMIS) remains challenging. Current methods for surgical error detection typically involve two parts: identifying gestures and then detecting errors within each gesture clip. These methods often overlook the rich contextual and semantic information inherent in surgical videos, with limited performance due to reliance on accurate gesture identification. Inspired by the chain-of-thought prompting in natural language processing, this letter presents a novel and real-time end-to-end error detection framework, Chain-of-Gesture (COG) prompting, integrating contextual information from surgical videos step by step. This encompasses two reasoning modules that simulate expert surgeons' decision-making: a Gestural-Visual Reasoning module using transformer and attention architectures for gesture prompting and a Multi-Scale Temporal Reasoning module employing a multi-stage temporal convolutional network with slow and fast paths for temporal information extraction. We validate our method on the JIGSAWS dataset and show improvements over the state-of-the-art, achieving 4.6% higher F1 score, 4.6% higher Accuracy, and 5.9% higher Jaccard index, with an average frame processing time of 6.69 milliseconds. This demonstrates our approach's potential to enhance RMIS safety and surgical education efficacy. The code is available at https://github.com/jinlab-imvr/Chain-of-Gesture.

Keywords: Aerial Systems: Mechanics and Control, Physical Human-Robot Interaction, Aerial Systems: Applications
Abstract: This paper presents a new cargo transportation solution based on physical human-robot interaction utilizing a novel fully-actuated multirotor platform called Palletrone. The platform is designed with a spacious upper flat surface for easy cargo loading, complemented by a rear-mounted handle reminiscent of a shopping cart. Flight trajectory control is achieved by a human operator gripping the handle and applying three-dimensional forces and torques while maintaining a stable cargo transport with zero roll and pitch attitude throughout the

flight. To facilitate physical human-robot interaction, we employ an admittance control technique. Instead of relying on complex force estimation methods, like in most admittance control implementations, we

introduce a simple yet effective estimation technique based on a disturbance observer robust control algorithm. We conducted an analysis

of the flight stability and performance in response to changes in system mass resulting from arbitrary cargo loading. Ultimately, we demonstrate that individuals can effectively control the system trajectory by applying appropriate interactive forces and torques. Furthermore, we showcase the performance of the system through various experimental scenarios.

Keywords: Aerial Systems: Mechanics and Control, Art and Entertainment Robotics, Tendon/Wire Mechanism
Abstract: Art is one of the oldest forms of human expression, constantly evolving, taking new forms and using new techniques. With their increased accuracy and versatility, robots can be considered as a new class of tools to perform works of art. The STRAD (STReet Art Drone) project aims to perform a 10-meter- high painting on a vertical surface with sub-centimetric precision. To achieve this goal we introduce a new design for an aerial manipulator with elastic suspension capable of moving from one equilibrium position to another using only its thrusters and an elliptic pulley-counterweight system. A feedback linearization control law is implemented to perform fast and accurate winding and unwinding of an elastic cable.

Keywords: Aerial Systems: Mechanics and Control, Aerial Systems: Applications, Mobile Manipulation
Abstract: Aerial physical interaction (APhI) with a multirotor-based platform such as pushing a heavy object demands generation of a sufficiently large interaction force while maintaining the stability. Such requirement can cause rotor saturation, because the rotor thrust enlarged for interaction force may leave a reduced margin for attitude stabilization. We first design an H -shaped coaxial tiltrotor that can generate a sufficiently large interaction force than a conventional multirotor. We then propose an overall framework composed of high-level robust controller and low-level control allocation for the coaxial tiltrotor to ensure robustness against uncertain motion of the unknown interacting object and to overcome the saturation issue. To guarantee the robustness at all time, we design a controller based on a nonlinear disturbance observer (DOB). Then, we formulate a problem of computing low-level actuator inputs avoiding rotor saturation as a tractable nonlinear optimization problem, which can be solved real-time. The proposed framework is validated in extensive real-world experiments where the 3.3 kg tiltrotor successfully pushes a cart weighing up to 60 kg. An ablation study with the tiltrotor shows effectiveness of the proposed control allocation law in avoiding rotor saturation. Furthermore, a comparative experiment with a conventional multirotor shows failure in the same setting, which validates the use of the coaxial tiltrotor. An experimental video can be found at htt

Keywords: Aerial Systems: Applications, Grasping, Motion Control
Abstract: Delivery by aerial robots is an emerging topic in many scenarios, such as logistics, construction industry, and disaster response. Compared to the standard styles that deploy cage or sling, grasping style by gripper can handle objects in various shapes. A multi-limbed structure with distributed vectorable rotors called SPIDAR shows a higher potential to grasp large object in a three-dimensional manner. Therefore, in this paper, we focus on the advanced usage of the vectored thrust forces to achieve aerial grasping by this robot. First, a vectored thrust control to avoid the aerointerference on the underwind segments (e.g., grasped object) during flight is proposed. Then, an optimization-based planning method that utilizes redundant vectored thrust forces for firm grasping is developed. Finally, we demonstrate the feasibility of the proposed flight control and grasp planning by performing challenging grasping and transporting motion with a spherical object of which the diameter is 0.6m. To the best of our knowledge, this work is the first to achieve multi-finger-like grasping to carry a large object in midair.

Keywords: Aerial Systems: Applications, Motion and Path Planning, Planning under Uncertainty
Abstract: This paper investigates payload grasping from a moving platform using a hook-equipped aerial manipulator. First, a computationally efficient trajectory optimization based on complementarity constraints is proposed to determine the optimal grasping time. To enable application in complex, dynamically changing environments, the future motion of the payload is predicted using a physics simulator-based model. The success of payload grasping under model uncertainties and external disturbances is formally verified through a robustness analysis method based on integral quadratic constraints. The proposed algorithms are evaluated in a high-fidelity physical simulator, and in real flight experiments using a custom-designed aerial manipulator platform.

Keywords: Aerial Systems: Applications, Aerial Systems: Mechanics and Control, Cooperating Robots, Physical Human-Robot Interaction
Abstract: Human-robot interaction will play an essential role in various industries and daily tasks, enabling robots to effectively collaborate with humans and reduce their physical workload. This paper proposes a novel approach for physical human- robot collaborative transportation and manipulation of a cable- suspended payload with multiple aerial robots. The proposed method enables smooth and intuitive interaction between the transported objects and a human worker. In the same time, we consider distance constraints during the operations by exploiting the internal redundancy of the multi-robot transportation system. We validate the approach through extensive simulation and real-world experiments. These include scenarios where the robot team assists the human in transporting and manipulating a load, or where the human helps the robot team navigate the environment. We experimentally demonstrate for the first time, to the best of our knowledge, that our approach enables a quadrotor team to physically collaborate with a human in manipulating a payload in all 6 DoF in collaborative human- robot transportation and manipulation tasks.

Keywords: Vision-Based Navigation
Abstract: Vision-and-Language Navigation in Continuous Environments (VLN-CE) requires agents to navigate with low-level actions following natural language instructions in 3D environments. Most existing approaches utilize observation features from the current step to represent the viewpoint. However, these representations often conflate redundant and essential information for navigation, introducing ambiguity into the agent's action prediction. To address the problem of inadequate representation, we propose a Knowledge-and-History Aware Visual Representation for Continuous Vision-and-Language Navigation (VLN-KHVR). The proposed approach constructs enriched visual representations tailored to navigation instructions, enhancing agents’ navigation performance. Specifically, VLN-KHVR extracts image features from the current observation, retrieves relevant knowledge in the knowledge base, and obtains the history of the navigation episode. Subsequently, the knowledge and history features are filtered to eliminate the information irrelevant to navigation instruction. These refined features are integrated with the instruction for further interaction. Finally, the aggregated features are used to guide navigation. Our model outperforms previous methods on the VLN-CE benchmark, demonstrating the effectiveness of the proposed method.

Keywords: Localization, Vision-Based Navigation, SLAM
Abstract: This paper presents LiteVLoc, a hierarchical vi-sual localization framework that uses a lightweight topo-metric map to represent the environment. The method consists of three sequential modules that estimate camera poses in a coarse-to-fine manner. Unlike dense 3D mapping methods, LiteVLoc reduces storage by avoiding geometric reconstruction. It uses a learning-based feature matcher to establish dense corre-spondences between sparse keyframes and observations, and then refines poses with a geometric solver, enabling robustness to viewpoint changes. The system assumes depth sensors or stereo camera for deployment. A novel dataset for the map-free relocalization task is also introduced. Extensive experiments including localization and navigation in both simulated and real-world scenarios have validate the system’s performance and demonstrated its precision and efficiency for large-scale de-ployment. Code and data will be made publicly available at the webpage: https://rpl-cs-ucl.github.io/LiteVLoc.

Keywords: Vision-Based Navigation, Search and Rescue Robots, Probabilistic Inference
Abstract: Image-goal navigation enables a robot to reach the location where a target image was captured, using visual cues for guidance. However, current methods either rely heavily on data and computationally expensive learning-based approaches or lack efficiency in complex environments due to insufficient exploration strategies. To address these limitations, we propose Bayesian Embodied Image-goal Navigation Using Gaussian Splatting, a novel method that formulates ImageNav as an optimal control problem within a model predictive control framework. BEINGS leverages 3D Gaussian Splatting as a scene prior to predict future observations, enabling efficient, real-time navigation decisions grounded in the robot’s sensory experiences. By integrating Bayesian updates, our method dynamically refines the robot's strategy without requiring extensive prior experience or data. Our algorithm is validated through extensive simulations and physical experiments, showcasing its potential for embodied robot systems in visually complex scenarios. Project Page: www.mwg.ink/BEINGS-web.

Keywords: View Planning for SLAM, Vision-Based Navigation, SLAM
Abstract: This paper presents FLAF, a focal line and feature-constrained active view planning method for tracking failure avoidance in feature-based visual navigation of mobile robots. FLAF is built on a feature-based visual teach and repeat (VT&R) framework, which supports robotic applications by teaching robots to cruise various paths that fulfill many daily autonomous navigation requirements. However, tracking failures in feature-based Visual Simultaneous Localization and Mapping (VSLAM), particularly in textureless regions common in human-made environments, poses a significant challenge to the real-world deployment of VT&R. To address this problem, the proposed view planner is integrated into a feature-based VSLAM system, creating an active VT&R solution that mitigates tracking failures. Our system features a Pan-Tilt Unit (PTU)-based active mounted on a mobile robot. Using FLAF, the active camera-based VSLAM (AC-SLAM) operates during the teaching phase to construct a complete path map and in the repeating phase to maintain stable localization. FLAF actively directs the camera toward more map points to avoid mapping failures during path learning and toward more feature-identifiable map points while following the learned trajectory. Experimental results in real scenarios show that FLAF significantly outperforms existing methods by accounting for feature identifiability, particularly the view angle of the features. While effectively dealing with low-texture regions in active view planning, considering feature identifiability enables our active VT&R system to perform well in challenging environments.

Keywords: Deep Learning Methods, Vision-Based Navigation
Abstract: Vision-and-Language Navigation (VLN) empowers agents to associate time-sequenced visual observations with corresponding instructions to make sequential decisions. However, dealing with visually diverse scenes or transitioning from simulated environments to real-world deployment is still challenging. In this paper, we address the mismatch between human-centric instructions and quadruped robots with a low-height field of view, proposing a Ground-level Viewpoint Navigation (GVNav) approach to mitigate this issue. This work represents the first attempt to highlight the generalization gap in VLN across varying heights of visual observation in realistic robot deployments. Our approach leverages weighted historical observations as enriched spatiotemporal contexts for instruction following, effectively managing feature collisions within cells by assigning appropriate weights to identical features across different viewpoints. This enables low-height robots to overcome challenges such as visual obstructions and perceptual mismatches. Additionally, we transfer the connectivity graph from the HM3D and Gibson datasets as an extra resource to enhance spatial priors and a more comprehensive representation of real-world scenarios, leading to improved performance and generalizability of the waypoint predictor in real-world environments. Extensive experiments demonstrate that our Ground-level Viewpoint Navigation (GVnav) approach significantly improves performance in both simulated environments and real-world deployments with quadruped robots.

Keywords: Vision-Based Navigation, Representation Learning, Reinforcement Learning
Abstract: This paper introduces Navigation Transformer (NavTr), a novel framework for object-goal navigation using Transformer queries to enhance the learning and representation of environment states. By integrating semantic information, object positions, and neighborhood information, NavTr creates a unified, comprehensive, and extensible state representation for the object-goal navigating task. In the framework, the Transformer queries implicitly learn inter-object relationships, which facilitates high-level understanding of the environment. Additionally, NavTr implements target-oriented supervisory signals, such as rotation rewards and spatial loss, which improve exploration efficiency in the reinforcement learning framework. NavTr outperforms popular graph-based and Attention-based methods by a large margin in terms of success rate (SR) and success weighted by path length (SPL). Extensive experiments on the AI2-THOR dataset demonstrate the effectiveness of our approach.

Keywords: Marine Robotics, Autonomous Vehicle Navigation, SLAM
Abstract: Existing bathymetric simultaneous localization and mapping (SLAM) methods predominantly rely on odometry information for loop closure detection, which has a deteriorating performance when handling unreliable odometry data or conducting large-scale mapping missions. This letter introduces a novel generalized Bag of Words (BoW) named Shape BoW (S-BoW) for appearance-based loop closure detection in bathymetric SLAM. S-BoW is trained from the collection of the terrain gradient features extracted from existing bathymetric datasets and can be used in various bathymetric scenarios. We integrated the loop closure detection method using S-BoW into a feature-based bathymetric SLAM method called TTT SLAM, and we evaluated its performance against three existing bathymetric SLAM methods using two datasets. The results indicate that S-BoW not only serves as a generalized BoW but also enhances the efficiency of the integrated SLAM method, achieving accuracy comparable to the original TTT SLAM while offering a 37% speed improvement in a large-scale sea trial dataset. To the best of our knowledge, S-BoW is the first generalized BoW that can be used to realize effective appearance-based loop closure detection in bathymetric SLAM.

Keywords: Marine Robotics, Recognition, Data Sets for Robot Learning
Abstract: Oysters are an important keystone species in coastal ecosystems that provide several economic, environmental, and cultural benefits. Given the array of utilities derived from oysters, the application of autonomous robotic systems for oyster detection and monitoring grows increasingly relevant. However, current monitoring strategies for assessing oyster assemblages are mostly destructive. While manually identifying and monitoring oysters from video footage is nondestructive, is it tedious and requires expert input.

An alternative to human monitoring is deploying trained object detection models on edge devices, such as the Aqua2 robot, to enable real-time monitoring of oysters directly in the field. Yet training these models to maximum efficacy requires an extensive dataset that accurately represents the domain, and it is difficult to obtain such high-quality training data due to the complications inherent to underwater environments. To address these complications, we introduce a novel method leveraging stable diffusion to generate high-quality synthetic data for the marine domain. We exploit diffusion models to create photorealistic oyster imagery, using ControlNet inputs to ensure consistency with the segmentation ground-truth mask, the geometry of the scene, and the target domain of real oyster images. This large dataset is used to train a vision model, specifically based on YOLOv10. The trained model is then deployed and tested on an edge platform, the Aqua2, in an underwater robotics system. We achieve state-of-the-art (0.657 mAP@50) for oyster detection, which can pave the way for autonomous oyster habitat monitoring and increase the efficiency of on-bottom oyster aquaculture

Keywords: Marine Robotics, Data Sets for Robotic Vision, Visual Learning
Abstract: We present an image blending pipeline, IBURD, that creates realistic synthetic images to assist in the training of deep detectors for use on underwater autonomous vehicles (AUVs) for marine debris detection tasks. Specifically, IBURD generates both images of underwater debris and their pixel-level annotations, using source images of debris objects, their annotations, and target background images of marine environments. With Poisson editing and style transfer techniques, IBURD is even able to robustly blend transparent objects into arbitrary backgrounds and automatically adjust the style of blended images using the blurriness metric of target background images. These generated images of marine debris in actual underwater backgrounds address the data scarcity and data variety problems faced by deep-learned vision algorithms in challenging underwater conditions, and can enable the use of AUVs for environmental cleanup missions. Both quantitative and robotic evaluations of IBURD demonstrate the efficacy of the proposed approach for robotic detection of marine debris.

Keywords: Marine Robotics, Mapping
Abstract: Underwater autonomous robotic operations require online localization and 3D mapping. Because of the absence of absolute positioning underwater, these tasks strongly rely on embedded sensors, including proprioceptive or navigation sensors — which can be fused for an odometry, — and exteroceptive sensors. One of the most popular exteroceptive sensors for underwater is the imaging sonar, which emits a large fan-shaped acoustic signal and estimates the position of the surrounding obstacles from a measure of the reflected signal. This paper addresses underwater online localization and 3D mapping using a forward looking, wide-aperture imaging sonar and vehicle’s intrinsic navigation estimates. We introduce 3DSSDF (3D Sonar Reconstruction Using Signed Distance Functions), a new localization and 3D mapping algorithm based on signed distance functions, which is evaluated in simulation and on real data, in man-made and natural environments. Comparisons to reference trajectories and maps demonstrate that, in our tests, 3DSSDF efficiently corrects navigation drift and that trajectory and map accuracy is always below 1 m and below 1% of the distanced travelled, which can be sufficient for the safe inspection of natural or artificial underwater structures.

Keywords: Marine Robotics, Localization, Sensor Fusion
Abstract: This paper presents a novel cascade nonlinear observer framework for inertial state estimation. It tackles the problem of intermediate state estimation when external localization is unavailable or in the event of a sensor outage. The proposed observer comprises two nonlinear observers based on a recently developed iteratively preconditioned gradient descent (IPG) algorithm. It takes the inputs via an IMU preintegration model where the first observer is a quaternion-based IPG. The output for the first observer is the input for the second observer, estimating the velocity and, consequently, the position. The proposed observer is validated on a public underwater dataset and a real-world experiment using our robot platform. The estimation is compared with an extended Kalman filter (EKF) and an invariant extended Kalman filter (InEKF). Results demonstrate that our method outperforms these methods regarding better positional accuracy and lower variance.

Keywords: Marine Robotics, Planning under Uncertainty, Collision Avoidance
Abstract: Motion planning for autonomous active perception in cluttered environments remains a challenging problem, requiring real-time solutions that both maximize safety and achieve a desired behavior. In dynamic underwater environments, such as in aquaculture operations, the robots are additionally expected to deal with state and motion uncertainty and errors, dynamic and deformable obstacles, currents, and disturbances. Previous work has introduced real-time frameworks that provided safe navigation in cluttered environments, active perception in static environments, and robust navigation in uncertain dynamic environments. This paper introduces a new real-time approach called ResiVis, which leverages the best aspects of the aforementioned techniques along with a new formulation that further enhances underwater autonomy by enabling active perception of static and dynamic target objects from desired distances. The proposed method utilizes path-optimization for real-time response with constraints guaranteeing continuous collision safety, and computes paths with clearance adaptive to both the conditions of the environments and the performance of the path follower. An improved new constraint encourages observations of dynamic objects with the planner adapting to satisfy desired observation distances and their projected future positions. ResiVis is validated with challenging simulation experiments and with hardware-in-the-loop trials in real industrial-scale aquaculture facilities.

Keywords: Legged Robots, Optimization and Optimal Control, Dynamics
Abstract: Symmetrical gaits, such as trotting, are com- monly employed in quadrupedal robots for their simplicity and stability. However, the potential of asymmetrical gaits, such as bounding and galloping—which are prevalent in their natural counterparts at high speeds or over long distances—is less clear in the design of locomotion controllers for legged machines. In these asymmetrical gaits, the system dynamics are more complex because the front and rear leg pairs exhibit different motions, which are coupled by the rotational motion of the torso. This study systematically examines five distinct asymmetrical quadrupedal gaits on a legged robot, aiming to uncover the fundamental differences in footfall sequences and the consequent energetics across a broad range of speeds. Utilizing a full-body model of a quadrupedal robot (Unitree A1), we developed a hybrid system for each gait, incorporating the desired footfall sequence and rigid impacts. To identify the most energy-optimal gait, we applied optimal control methods, framing it as a trajectory optimization problem with specific constraints and a work-based cost of transport as an objective function. Our results show that, in the context of asymmetrical gaits, when minimizing cost of transport across the entire stride, the front leg pair primarily propels the system forward, while the rear leg pair acts more like an inverted pendulum, contributing significantly less to the energetic output. Addi- tionally, while bounding—characterized by two aerial phases per cycle—is the most energy-optimal gait at higher speeds, the energy expenditure of gaits at speeds below 1 m/s depend heavily on the robot’s specific design.

Keywords: Legged Robots, Compliant Joints and Mechanisms, Motion Control
Abstract: In biomechanics and robotics, elasticity plays a crucial role in enhancing locomotion efficiency and stability. Traditional approaches in legged robots often employ series elastic actuators (SEA) with discrete rigid components, which, while effective, add weight and complexity. This paper presents an innovative alternative by integrating continuously compliant structures into the lower legs of a bipedal robot, fundamentally transforming the SEA concept. Our approach replaces traditional rigid segments with lightweight, deformable materials, reducing overall mass and simplifying the actuation design. This novel design introduces unique challenges in modeling, sensing, and control, due to the infinite dimensionality of continuously compliant elements. We address these challenges through effective approximations and control strategies. The paper details the design and modeling of the compliant leg structure, presents low-level force and kinematics controllers, and introduces a high-level posture controller with a gait scheduler. Experimental results demonstrate successful bipedal walking using this new design.

Keywords: Legged Robots, Robust/Adaptive Control, Multi-Contact Whole-Body Motion Planning and Control
Abstract: As legged robots continue to evolve, new control methods are being developed to provide fast, robust, accurate and computationally efficient algorithms for traversing challenging environments. This paper presents a real-time adaptive locomotion controller for quadrupeds, designed to maintain stability and controllability on various surfaces, including highly slippery terrains. The proposed approach optimizes control effort distribution based on the probability of slippage by utilizing a surface-independent adaptation layer. By balancing the robot's redundant kinematic system through rank relaxation—similar to loosening constraints in optimization problems—this method demonstrates significant performance improvements. Unlike Reinforcement Learning (RL) approaches, which depend on pre-trained policies and may struggle to adapt velocity tracking control across different terrains, our method rapidly adjusts to changing conditions, as validated by extensive simulation experiments.

Keywords: Legged Robots, Reinforcement Learning, Natural Machine Motion
Abstract: In reinforcement learning for legged robot locomotion, crafting effective reward strategies is crucial. Predefined gait patterns and complex reward systems are widely used to stabilize policy training. Drawing from the natural locomotion behaviors of humans and animals, which adapt their gaits to minimize energy consumption, we investigate the impact of incorporating an energy-efficient reward term that prioritizes distance-averaged energy consumption into the reinforcement learning framework. Our findings demonstrate that this simple addition enables quadruped robots to autonomously select appropriate gaits—such as four-beat walking at lower speeds and trotting at higher speeds—without the need for explicit gait regularizations. Furthermore, we provide a guideline for tuning the weight of this energy-efficient reward, facilitating its application in real-world scenarios. The effectiveness of our approach is validated through simulations and on a real Unitree Go1 robot. This research highlights the potential of energy-centric reward functions to simplify and enhance the learning of adaptive and efficient locomotion in quadruped robots. Videos and more details are at https://sites.google.com/berkeley.edu/efficient-locomotion.

Keywords: Legged Robots, Reinforcement Learning, Biomimetics
Abstract: We address the challenge of effectively controlling the locomotion of legged robots by incorporating precise frequency and phase characteristics, which is often ignored in locomotion policies that do not account for the periodic nature of walking. We propose a hierarchical architecture that integrates a low-level phase tracker, oscillators, and a high-level phase modulator. This controller allows quadruped robots to walk in a natural manner that is synchronized with external musical rhythms. Our method generates diverse gaits across different frequencies and achieves real-time synchronization with music in the physical world. This research establishes a foundational framework for enabling real-time execution of accurate rhythmic motions in legged robots. The video and code are available at https://music-walker.github.io/.

Keywords: Humanoid Robot Systems, Sensorimotor Learning, Representation Learning
Abstract: Humanoid robots require both robust lower-body locomotion and precise upper-body manipulation. While recent Reinforcement Learning (RL) approaches provide whole-body loco-manipulation policies, they lack precise manipulation with high DoF arms. In this paper, we propose decoupling upper-body control from locomotion, using inverse kinematics (IK) and motion retargeting for precise manipulation, while RL focuses on robust lower-body locomotion. We introduce PMP (Predictive Motion Priors), trained with Conditional Variational Autoencoder (CVAE) to effectively represent upper-body motions. The locomotion policy is trained and conditioned on this upper-body motion representation, ensuring that the system remains robust with both manipulation and locomotion. We show that CVAE features are crucial for stability and robustness, and significantly outperforms RL-based whole-body control in precise manipulation. With precise upper-body motion and robust lower-body locomotion control, operators can remotely control the humanoid to walk around and explore different environments, while performing diverse manipulation tasks.

Keywords: Multi-Robot Systems, Collision Avoidance, Optimization and Optimal Control
Abstract: This paper proposes a distributed controller synthesis framework for safe navigation of multi-agent systems. We leverage control barrier functions to formulate collision avoidance with obstacles and teammates as constraints on the control input for a state-dependent network optimization problem that encodes team formation and the navigation task. Our algorithmic solution is valid under general assumptions for nonlinear dynamics and state-dependent network optimization problems with convex constraints and strongly convex objectives. The resulting controller is distributed, satisfies the safety constraints at all times, and asymptotically converges to the solution of the state-dependent network optimization problem. We illustrate its performance in a team of differential-drive robots in a variety of complex environments, both in simulation and in hardware.

Keywords: Multi-Robot Systems, Cooperating Robots, Constrained Motion Planning
Abstract: This paper presents a framework for multi-agent navigation in structured but dynamic environments, integrating three key components: a shared semantic map encoding metric and semantic environmental knowledge, a claim policy for coordinating access to areas within the environment, and a Model Predictive Controller for generating motion trajectories that respect environmental and coordination constraints. The main advantages of this approach include: (i) enforcing area occupancy constraints derived from specific task requirements; (ii) enhancing computational scalability by eliminating the need for collision avoidance constraints between robotic agents; and (iii) the ability to anticipate and avoid deadlocks between agents. The paper includes both simulations and physical experiments demonstrating the framework’s effectiveness in various representative scenarios

Keywords: Aerial Systems: Applications, Distributed Robot Systems, Collision Avoidance
Abstract: Multi-quadrotor systems face significant challenges in decentralized control, particularly with safety and coordination under sensing and communication limitations. State-of-the-art methods leverage Control Barrier Functions (CBFs) to provide safety guarantees but often neglect actuation constraints and limited detection range. To address these gaps, we propose a novel decentralized Nonlinear Model Predictive Control (NMPC) that integrates Exponential CBFs (ECBFs) to enhance safety and optimality in multi-quadrotor systems. We provide both conservative and practical minimum bounds of the range that preserve the safety guarantees of the ECBFs. We validate our approach through extensive simulations with up to 10 quadrotors and 20 obstacles, as well as real-world experiments with 3 quadrotors. Results demonstrate the effectiveness of the proposed framework in realistic settings, highlighting its potential for reliable quadrotor teams operations.

Keywords: Deep Learning Methods, Path Planning for Multiple Mobile Robots or Agents, Reinforcement Learning
Abstract: The Multi-Agent Path Finding (MAPF) problem aims to determine the shortest and collision-free paths for multiple agents in a known, potentially obstacle-ridden environment. It is the core challenge for robotic deployments in large-scale logistics and transportation. Decentralized learning-based approaches have shown great potential for addressing the MAPF problems, offering more reactive and scalable solutions. However, existing learning-based MAPF methods usually rely on agents making decisions based on a limited field of view (FOV), resulting in short-sighted policies and inefficient cooperation in complex scenarios. There, a critical challenge is to achieve consensus on potential movements between agents based on limited observations and communications. To tackle this challenge, we introduce a new framework that applies sheaf theory to decentralized deep reinforcement learning, enabling agents to learn geometric cross-dependencies between each other through local consensus and utilize them for tightly cooperative decision-making. In particular, sheaf theory provides a mathematical proof of conditions for achieving global consensus through local observation. Inspired by this, we incorporate a neural network to approximately model the consensus in latent space based on sheaf theory and train it through self-supervised learning. During the task, in addition to normal features for MAPF as in previous works, each agent distributedly reasons about a learned consensus feature, leading to efficient cooperation on pathfinding and collision avoidance. As a result, our proposed method demonstrates significant improvements over state-of-the-art learning-based MAPF planners, especially in relatively large and complex scenarios, demonstrating its superiority over baselines in various simulations and real-world robot experiments.

Keywords: Path Planning for Multiple Mobile Robots or Agents, Multi-Robot Systems, Distributed Robot Systems
Abstract: This work presents an algorithm based on the Nondominated Sorting Genetic Algorithm II (NSGA-II) to solve multi-objective offline Multi-Robot Coverage Path Planning (MCPP) problems. The proposed algorithm embeds a donation-mutation operator and a multiple-parent crossover that generates solutions which maintain the longest path while minimizing the average path length. The algorithm also uses a library of elitism-selected high-fitness robot paths, and tournament-selected high min-max fitness paths, to construct high multi-objective fitness offspring. We evaluate the performance of our proposed algorithm against the state-of-the-art NSGA-II extended with an improved Heuristic Genetic Algorithm Crossover, and we demonstrate that for different instances of the MCPP problem, the Pareto-fronts of our proposed algorithm are not dominated by any of the points of the fronts generated by the state-of-the-art NSGA-II. A comparison has also been performed in a virtual environment simulating five drones inspecting three wind turbines. Results show that our approach exhibits a higher convergence rate for higher values of the ratio between the number of points to visit and the number of drones.

Keywords: Formal Methods in Robotics and Automation, Planning, Scheduling and Coordination, Multi-Robot Systems
Abstract: This paper presents an iterative approach for heterogeneous multi-agent route planning in environments with unknown resource distributions. We focus on a team of robots with diverse capabilities tasked with executing missions specified using Capability Temporal Logic (CaTL), a formal framework built on Signal Temporal Logic to handle spatial, temporal, capability, and resource constraints. The key challenge arises from the uncertainty in the initial distribution and quantity of resources in the environment. To address this, we introduce an iterative algorithm that dynamically balances exploration and task fulfillment. Robots are guided to explore the environment, identifying resource locations and quantities while progressively refining their understanding of the resource landscape. At the same time, they aim to maximally satisfy the mission objectives based on the current information, adapting their strategies as new data is uncovered. This approach provides a robust solution for planning in dynamic, resource-constrained environments, enabling efficient coordination of heterogeneous teams even under conditions of uncertainty. Our method's effectiveness and performance are demonstrated through simulated case studies.

Keywords: Path Planning for Multiple Mobile Robots or Agents, Integrated Planning and Learning, Distributed Robot Systems
Abstract: This letter addresses the distributed informative path planning (IPP) problem for a mobile robot network to optimally explore a spatial field. Each robot is able to gather noisy environmental measurements while navigating the environment and build its own model of a spatial phenomenon using the Gaussian process and local data. The IPP optimization problem is formulated in an informative way through a multi-step prediction scheme constrained by connectivity preservation and collision avoidance. The shared hyperparameters of the local Gaussian process models are also arranged to be optimally computed in the path planning optimization problem. By the use of the proximal alternating direction method of multiplier, the optimization problem can be effectively solved in a distributed manner. It theoretically proves that the connectivity in the network is maintained over time whilst the solution of the optimization problem converges to a stationary point. The effectiveness of the proposed approach is verified in synthetic experiments by utilizing a real-world dataset.

Keywords: Path Planning for Multiple Mobile Robots or Agents, Planning, Scheduling and Coordination, Task Planning
Abstract: The Multiple Depot Traveling Salesman Problem (MDTSP) is a variant of the NP-hard Traveling Salesman Problem (TSP) with more than one salesman to jointly visit all destinations, commonly found in task planning in multi-agent robotic systems. Traditional MDTSP overlooks practical constraints like limited battery level and inter-agent conflicts, often leading to infeasible or unsafe solutions in reality. In this work, we incorporate energy and resource consumption constraints to form the Constrained MDTSP (CMDTSP). We design a novel hierarchical framework to obtain high-quality solutions with low computational complexity. The framework decomposes a given CMDTSP instance into manageable sub-problems, each handled individually via a TSP solver and heuristic search to generate tours. The tours are then aggregated and processed through a Mixed-Integer Linear Program (MILP), which contains significantly fewer variables and constraints than the MILP for the exact CMDTSP, to form a feasible solution efficiently. We demonstrate the performance of our framework on both real-world and synthetic datasets. It reaches a mean 12.48% optimality gap and 41.7x speedup over the exact method on common instances and a 5.22%sim14.84% solution quality increase with more than 79.8x speedup over the best baseline on large instances where the exact method times out.

Keywords: Path Planning for Multiple Mobile Robots or Agents, Environment Monitoring and Management, Integrated Planning and Learning
Abstract: Autonomous robots can survey and monitor large environments. However, these robots often have limited computational and power resources, making it crucial to develop an efficient and adaptive informative path planning (IPP) algorithm. Such an algorithm must quickly adapt to environmental data to maximize the information collected while accommodating path constraints, such as distance budgets and boundary limitations.

Current approaches to this problem often rely on maximizing mutual information using methods such as greedy algorithms, Bayesian optimization, and genetic algorithms. These methods can be slow and do not scale well to large or 3D environments. We present an adaptive IPP approach that is fully differentiable, significantly faster than previous methods, and scalable to 3D spaces. Our approach also supports continuous sensing robots, which collect data continuously along the entire path, by leveraging streaming sparse Gaussian processes.

Benchmark results on two real-world datasets demonstrate that our approach yields solutions that are on par with or better than baseline methods while being up to two orders of magnitude faster. Additionally, we showcase our adaptive IPP approach in a 3D space using a system-on-chip embedded computer with minimal computational resources. Our code is available in the SGP-Tools Python library with a companion ROS 2 package for deployment on ArduPilot-based robots.

Keywords: Environment Monitoring and Management, Motion and Path Planning, Reactive and Sensor-Based Planning
Abstract: Information gathering focuses on designing strategies for a robot to collect data about a physical process, aiming for accurate field reconstruction. While many recent methods have been proposed to address this problem, they often assume the model of the physical process is a priori known and stationary—assumptions that rarely hold in practice. This paper presents a novel informative motion planning approach for online information gathering of a non-stationary Gaussian process. Our approach comprises two key components: an informative path planner that explores the physical field and an adaptive velocity planner that adjusts the robot's velocity profile exploiting the field's spatial variability. Additionally, we propose a path smoothing and tracking strategy to ensure continuous robot motion. Extensive simulations on a bathymetric mapping task demonstrate the effectiveness of our approach, showing superior performance in reconstructing non-stationary physical fields compared to several baseline methods.

Keywords: Search and Rescue Robots, Path Planning for Multiple Mobile Robots or Agents, Robotics in Under-Resourced Settings
Abstract: Smart factories enhance production efficiency and sustainability, but emergencies like human errors, machinery failures and natural disasters pose significant risks. In critical situations, such as fires or earthquakes, collaborative robots can assist first-responders by entering damaged buildings and locating missing persons, mitigating potential losses. Unlike previous solutions that overlook the critical aspect of energy management, in this paper we propose REACT, a smart energy-aware orchestrator that optimizes the exploration phase, ensuring prolonged operational time and effective area coverage. Our solution leverages a fleet of collaborative robots equipped with advanced sensors and communication capabilities to explore and navigate unknown indoor environments, such as smart factories affected by fires or earthquakes, with high density of obstacles. By leveraging real-time data exchange and cooperative algorithms, the robots dynamically adjust their paths, minimize redundant movements and reduce energy consumption. Extensive simulations confirm that our approach significantly improves the efficiency and reliability of search and rescue missions in complex indoor environments, improving the exploration rate by 10% over existing methods and reaching a map coverage of 97% under time critical operations, up to nearly 100% under relaxed time constraint.

Keywords: Aerial Systems: Perception and Autonomy, Vision-Based Navigation, Path Planning for Multiple Mobile Robots or Agents
Abstract: In this work, we consider the problem of multi-agent informative path planning (IPP) for robots whose sensor visibility evolves over time as a consequence of a time-varying natural phenomenon. We leverage ergodic trajectory optimization (ETO), which generates paths such that the amount of time an agent spends in an area is proportional to the expected information in that area. We focus specifically on the problem of multi-agent drone search of a wildfire, where we use the time-varying environmental process of smoke diffusion to construct a sensor visibility model. This sensor visibility model is used to repeatedly calculate an expected information distribution (EID) to be used in the ETO algorithm. Our experiments show that our exploration method achieves improved information gathering over both baseline search methods and naive ergodic search formulations.

Keywords: Optimization and Optimal Control, Machine Learning for Robot Control, Model Learning for Control
Abstract: Model predictive control (MPC) has been applied to many platforms in robotics and autonomous systems for its capability to predict a system's future behavior while incorporating constraints that a system may have. To enhance the performance of a system with an MPC controller, one can manually tune the MPC's cost function. However, it can be challenging due to the possibly high dimension of the parameter space as well as the potential difference between the open-loop cost function in MPC and the overall closed-loop performance metric function. This paper presents DiffTune-MPC, a novel learning method, to learn the cost function of an MPC in a closed-loop manner. The proposed framework is compatible with the scenario where the time interval for performance evaluation and MPC's planning horizon have different lengths. We show the auxiliary problem whose solution admits the analytical gradients of MPC and discuss its variations in different MPC settings, including nonlinear MPCs that are solved using sequential quadratic programming. Simulation results demonstrate the learning capability of DiffTune-MPC and the generalization capability of the learned MPC parameters.

Keywords: Robot Safety, Motion Control, Robust/Adaptive Control
Abstract: 轮式自重构机器人 （WSRRs） 是一种新型的多机器人系统，具有灵活的配置和任务适应性，在非结构化任务环境中具有广泛的应用前景。该文基于非完整约束和拉格朗日方法，建立了具有任意重配置尺度的 WSRR 的组合模态运动学和动力学。在运动学层面，基于非完整约束，设计了基于安全地理围栏的平滑避障策略来确保安全。在动态层面，引入自适应容错机制，保证合理的扭矩分配，避免跟踪性能下降。同时，该文阐述了一种改进的扩展状态观测器（IESO），通过该算法可以抑制测量噪声的高频振荡和初始观测器误差的峰值现象，实现了未知集总扰动下鲁棒的速度跟踪控制

Keywords: Robot Safety, Intelligent Transportation Systems, Autonomous Vehicle Navigation
Abstract: Simulation is an indispensable tool in the development and testing of autonomous vehicles (AVs), offering an efficient and safe alternative to road testing. An outstanding challenge with simulation-based testing is the generation of safety-critical scenarios, which are essential to ensure that AVs can handle rare but potentially fatal situations. This paper addresses this challenge by introducing a novel framework, CaDRE, to generate realistic, diverse, and controllable safety-critical scenarios. Our approach optimizes for both the quality and diversity of scenarios by employing a unique formulation and algorithm that integrates real-world scenarios, domain knowledge, and black-box optimization. We validate the effectiveness of our framework through extensive testing in three representative types of traffic scenarios. The results demonstrate superior performance in generating diverse and high-quality scenarios with greater sample efficiency than existing reinforcement learning (RL) and sampling-based methods.

Keywords: Robot Safety, Reinforcement Learning, Machine Learning for Robot Control
Abstract: Control Barrier Functions (CBFs) have emerged as a prominent approach to designing safe navigation systems of robots. Despite their popularity, current CBF-based methods exhibit some limitations: optimization-based safe control techniques tend to be either myopic or computationally intensive, and they rely on simplified system models; conversely, the learning-based methods suffer from the lack of quantitative indication in terms of navigation performance and safety. In this paper, we present a new model-free reinforcement learning algorithm called Certificated Actor-Critic (CAC), which introduces a hierarchical reinforcement learning framework and well-defined reward functions derived from CBFs. We carry out theoretical analysis and proof of our algorithm, and propose several improvements in algorithm implementation. Our analysis is validated by two simulation experiments, showing the effectiveness of our proposed CAC algorithm.

Keywords: Robot Safety, Machine Learning for Robot Control
Abstract: Hamilton-Jacobi (HJ) reachability analysis is a widely adopted verification tool to provide safety and performance guarantees for autonomous systems. However, it involves solving a partial differential equation (PDE) to compute a safety value function, whose computational and memory complexity scales exponentially with the state dimension, making its direct application to large-scale systems intractable. To overcome these challenges, DeepReach,a recently proposed learning-based approach, approximates high-dimensional reachable tubes using neural networks (NNs). While shown to be effective, the accuracy of the learned solution decreases with system complexity. One of the reasons for this degradation is a soft imposition of safety constraints during the learning process, which corresponds to the boundary conditions of the PDE, resulting in inaccurate value functions. In this work, we propose ExactBC, a variant of DeepReach that imposes safety constraints exactly during the learning process by restructuring the overall value function as a weighted sum of the boundary condition and the NN output. Moreover, the proposed variant no longer needs a boundary loss term during the training process, thus eliminating the need to balance different loss terms. We demonstrate the efficacy of the proposed approach in significantly improving the accuracy of the learned value function for four challenging reachability tasks: a rimless wheel system with state resets, collision avoidance in a cluttered environment, autonomous rocket landing, and multi-aircraft collision avoidance.

Keywords: Robot Safety, AI-Based Methods, Probability and Statistical Methods
Abstract: AI-controlled robotic systems can introduce significant risks to both humans and the environment. Traditional reliability assessment methods fall short in addressing the complexities of these systems, particularly when dealing with black-box or dynamically changing control policies. The traditional approaches are applied manually and do not consider frequent software updates. In this paper, we present RelAIBotiX, a new methodology that enables dynamic and continuous reliability assessment, specifically tailored for robotic systems controlled by AI-Algorithms. RelAIBotiX is a dynamic reliability assessment framework that combines four methods: (i) Skill Detection that automatically identifies executed skills using deep learning techniques, (ii) Behavioral Analysis that creates an operational profile of the robotic system containing information about the skill execution sequence, active components for each skill, and their utilization intensity that influence their failure rate, (iii) Reliability Model Generation that automatically transforms the operational profile and reliability data of robotic hardware components into quantitative hybrid reliability models, and (iv) Reliability Model Solver for the numerical evaluation of the generated reliability models. Our evaluation included computing the reliability of the system, the probability of failure of individual skills, and component sensitivity analysis. We validated the applicability of the proposed framework in five simulative and real-world setups.

Keywords: Emotional Robotics, Gesture, Posture and Facial Expressions, Robot Companions
Abstract: Social robots have been extensively studied in recent decades, with many researchers exploring the use of modalities such as facial expressions to achieve more natural emotions in robots. Various methods have been attempted to generate and express robot emotions, including computational models that define an affect space and show dynamic emotion changes. However, the implementation of multimodal expression in previous models is ambiguous, and the generation of emotions in response to stimuli relies on heuristic methods. In this paper, we present a framework that enables robots to naturally express their emotions in a multimodal way, where the emotion can change over time based on the given stimulus values. By representing the robot’s emotion as a position in an affect space of a computational emotion model, we consider the given stimuli values as driving forces that can shift the emotion position dynamically. In order to examine the feasibility of our proposed method, a mobile robot prototype was implemented that can recognize touch and express different emotions with facial expressions and movements. The experiment demonstrated that the emotion elicited by a given stimulus is contingent upon the robot’s previous state, thereby imparting the impression that the robot possesses a distinctive emotion model. Furthermore, the Godspeed survey results indicated that our model was rated significantly higher than the baseline, which did not include a computational emotion model, in terms of anthropomorphism, animacy, and perceived intelligence. Notably, the unpredictabil ity of emotion switching contributed to a perception of greater lifelikeness, which in turn enhanced the overall interaction experience.

Keywords: Human-Robot Collaboration, Multi-Modal Perception for HRI, Embedded Systems for Robotic and Automation
Abstract: Intelligent virtual agents are used to accomplish complex multi-modal tasks such as human instruction comprehension in mixed-reality environments by increasingly adopting richer, energy-intensive sensors and processing pipelines. In such applications, the context for activating sensors and processing blocks required to accomplish a given task instance is usually manifested via multiple sensing modes. Based on this observation, we introduce a novel Commit-and-Switch (CAS) paradigm that simultaneously seeks to reduce both sensing and processing energy. In CAS, we first commit to a low-energy computational pipeline with a subset of available sensors. Then, the task context estimated by this pipeline is used to optionally switch to another energy-intensive DNN pipeline and activate additional sensors. We demonstrate how CAS’s paradigm of interweaving DNN computation and sensor triggering can be instantiated principally by constructing multi-head DNN models and jointly optimizing the accuracy and sensing costs associated with different heads. We exemplify CAS via the development of the RealGIN-MH model for multi-modal target acquisition tasks, a core enabler of immersive human-agent interaction. RealGIN-MH achieves 12.9x reduction in energy overheads, while outperforming baseline dynamic model optimization approaches.

Keywords: Social HRI, Physical Human-Robot Interaction
Abstract: Humor is a key element in human interactions, essential for building connections and rapport. To enhance human-robot communication, we developed a humor-aware chat framework that enables robots to deliver contextually appropriate humor. This framework takes into account the interaction environment, and user’s profile as well as emotional state. Two GPT models are used to generate responses. The initial one, named sensor-GPT, processes contextual data from the sensor along with the user’s response and conversation history to create prompts for the second one, chat-GPT. These prompts can guide the model on how to integrate appropriate humor elements into the conversation, ensuring that the dialogue is both contextually relevant and humorous. Our experiment compared the effectiveness of humor expression between our framework and the GPT-4o model. The results demonstrate that robots using our framework significantly outperform those using GPT-4o in humor expression, extending conversations, and improving overall interaction quality.

Keywords: Physical Human-Robot Interaction, Touch in HRI
Abstract: Humans are able to convey different messages using only touch. Equipping robots with the ability to understand social touch adds another modality in which humans and robots can communicate. In this paper, we present a social gesture recognition system using a fabric-based, large-scale tactile sensor integrated onto the arms of a humanoid robot. We built a social gesture dataset using multiple participants and extracted temporal features for classification. By collecting real-world data on a humanoid robot, our system provides valuable insights into human-robot social touch, further advancing the development of spHRI systems for more natural and effective communication.

Keywords: Gesture, Posture and Facial Expressions, Intention Recognition, Human-Robot Collaboration
Abstract: Mobile manipulators - robots with a moving base and an arm for grasping objects - are becoming more common in human-populated environments, such as hospitals, warehouses, and even homes. Yet most mobile manipulators lack clear ways to communicate intent to human interlocutors in a continuous, socially acceptable, and easy-to-interpret way. One possible solution for improving mobile manipulator communication is the addition of expressive eyes. This paper presents the design and evaluation of a custom expressive LED eye module for mobile manipulators, which can display both gaze and emotional expressions. Our evaluation study (N = 32) involved a mock teamwork task alongside a Hello Robot Stretch RE2 mobile manipulator with the custom LED eye module. The results showed that both gaze and emotional expressions supported better participant performance in the task and more feelings of social closeness. Emotional eye expressions also yielded higher ratings of robot social warmth and competence. This work can inform mobile manipulator design for smoother integration into human-populated spaces.

Keywords: Social HRI, Gesture, Posture and Facial Expressions, Emotional Robotics
Abstract: Equipping humanoid robots with the capability to understand emotional states of human interactants and express emotions appropriately according to situations is essential for affective human-robot interaction. However, enabling current vision-aware multimodal emotion recognition models for affective human-robot interaction in the real-world raises embodiment challenges: addressing the environmental noise issue and meeting real-time requirements. First, in multiparty conversation scenarios, the noises inherited in the visual observation of the robot, which may come from either 1)distracting objects in the scene or 2) inactive speakers appearing in the field of view of the robot, hinder the models from extracting emotional cues from vision inputs. Secondly, realtime response, a desired feature for an interactive system, is also challenging to achieve. To tackle both challenges, we introduce an affective human-robot interaction system called UGotMe designed specifically for multiparty conversations. Two denoising strategies are proposed and incorporated into the system to solve the first issue. Specifically, to filter out distracting objects in the scene, we propose extracting face images of the speakers from the raw images and introduce a customized active face extraction strategy to rule out inactive speakers. As for the second issue, we employ efficient data transmission from the robot to the local server to improve realtime response capability. We deploy UGotMe on a human robot named Ameca to validate its real-time inference capabilities in practical scenarios. Videos demonstrating real-world deployment are available at https://lipzh5.github.io/HumanoidVLE/

Keywords: Soft Robot Applications, Soft Robot Materials and Design, Robotics and Automation in Life Sciences
Abstract: Automating small-scale experiments in materials science presents challenges due to the heterogeneous nature of experimental setups. This study introduces the SCU-Hand (Soft Conical Universal Robot Hand), a novel end-effector designed to automate the task of scooping powdered samples from various container sizes using a robotic arm. The SCU-Hand employs a flexible, conical structure that adapts to different container geometries through deformation, maintaining consistent contact without complex force sensing or machine learning-based control methods. Its reconfigurable mechanism allows for size adjustment, enabling efficient scooping from diverse container types. By combining soft robotics principles with a sheet-morphing design, our end-effector achieves high flexibility while retaining the necessary rigidity for effective powder manipulation. We detail the design principles, fabrication process, and experimental validation of the SCU-Hand. Experimental validation showed that the scooping capacity is about 20% higher than that of a commercial tool, with a scooping performance of more than 95% for containers of sizes between 67 mm to 110 mm. This research contributes to laboratory automation by offering a cost-effective, easily implementable solution for automating tasks such as materials synthesis and characterization processes.

Keywords: Compliant Joints and Mechanisms, Multifingered Hands
Abstract: Soft robotic hands compensate for uncertainty in perception and actuation by leveraging passive deformation in their intrinsically compliant hardware, facilitating robust and dexterous interactions with their environment. The ability to adjust the level of compliance during operation has the potential to further improve the performance of these hands by enabling novel interaction strategies. However, achieving variable stiffness mechanically typically requires significant engineering complexity, making these systems difficult to manufacture, prone to error, and expensive. We present a novel, very simple mechanism for achieving variable stiffness. This mechanism employs tendon-driven antagonistic actuation, with Bowden cables connecting elastic elements to servomotors. It supports compact actuator designs, while the Bowden cables facilitate flexible component placement within a robotic system. Following our approach, variable stiffness actuators can be easily manufactured at low-cost from readily available materials. Despite its simplicity, we demonstrate that our mechanism provides consistent and precise control over stiffness levels and contact torques, showcasing its potential for a broad range of applications in soft robotic systems.

Keywords: Soft Robot Applications, Physical Human-Robot Interaction, Grippers and Other End-Effectors
Abstract: This paper presents a novel design concept of a pair of soft gripper hands that can establish a secure connection between them for bearing a large load with a low air pressure. The design was inspired by dovetail joints in carpentry that enable a tight, strong connection between two pieces of wood. We propose to mimic the dovetail joint mechanism by using soft robotic fingers that interlace to each other for secure connection. The work was motivated by the need for securing a connection between two soft robotic arms for holding a balance-impaired older adult in case of losing balance. First, the design principle of dovetail-like secure soft finger connection is presented, and its potential application to a portable fall prevention system is described. Details of the dovetail soft finger design, its rapid inflation method, and other implementation issues are then discussed. Through experiments of a proof-of-concept prototype, it is validated that the dovetail soft fingers can bear at least 18 kg of load with only 52 kPa of air chamber pressure filled in 250 ms of charging time. At the end, the proposed method is compared to alternative methods using a Pugh chart.

Keywords: Soft Robot Applications, Grippers and Other End-Effectors, Grasping
Abstract: For objects with complex topological and geometrical features, stochastic topological grasping can be executed without the necessity for feedback or precise planning. However, this grasping method has two significant limitations. First, the technique’s effectiveness is reduced when interacting with topologically and geometrically simple objects like spheres, cubes, and cylinders, due to the inherent variability in grasping patterns. Additionally, the method’s low stiffness restricts its ability to securely handling heavier objects. To address these challenges, this paper proposes an entanglement soft robotic gripper with variable stiffness and two transformed grasping modes (entanglement and clamping modes). The gripper contains three filaments, which can enhance the stiffness through the mechanism of layer jamming. Furthermore, the entanglement mode and the clamping mode, can be transformed by adjusting the working length of the filaments. The grasping performance comparison with and without variable stiffness was carried out, and the results indicated that the implementation of variable stiffness led to a 149 % increase in payload weight. Through experimental validation, we successfully employed the gripper in variable stiffness and transformed modes to grasp items with various shapes and weights. Demonstration of grasping heavier objects and transforming between two grasping modes were also conducted to showcase the adaptability and versatility of the gripper.

Keywords: In-Hand Manipulation, Modeling, Control, and Learning for Soft Robots, Dexterous Manipulation
Abstract: Dynamic in-hand manipulation remains a challenging task for soft robotic systems that have demonstrated advantages in safe compliant interactions but struggle with high-speed dynamic tasks. In this work, we present SoftSpin, a system for dynamic spinning using a soft and compliant robotic hand. Unlike previous works that rely on quasi-static actions and precise object models, the proposed system learns to spin a pen through trial-and-error using only real-world data without requiring explicit prior knowledge of the pen’s physical attributes. With self-labeled trials sampled from the real world, the system discovers the set of pen grasping and spinning primitive parameters that enables a soft hand to spin the pen robustly and reliably. After 130 sampled actions, SoftSpin achieves 100 % success rate across three pens with different weights and weight distributions, demonstrating the system’s generalizability and robustness to changes in object properties. The results highlight the potential for soft robotic end-effectors to perform dynamic tasks including rapid in-hand manipulation. We also demonstrate that SoftSpin generalizes to spinning tools with different shapes and weights such as a brush and a screwdriver which we spin with 10/10 and 5/10 success rates respectively. Videos, data, and code are available at https://soft-spin.github.io

Keywords: Soft Robot Applications, Modeling, Control, and Learning for Soft Robots, Grippers and Other End-Effectors
Abstract: Soft robots (SR) are a class of continuum robots that enable safe human interaction with task versatility beyond rigid robots. This has resulted in their rapid adoption in a number of applications that require manipulation of delicate and irregular objects. Despite their advantages, SR grippers typically require case-specific experimental characterization for shape and gripper retention force estimation. This letter presents a kinetostatic modeling approach based on strain energy minimization subject to mechanics and geometric constraints for shape estimation of SR grippers with external tendon routing (ETR), including those with composite structures. Additionally, Castigliano's First Theorem is used to estimate the retention force of the gripper. These models are evaluated across four different ETR SR grippers. The mechanics model predicted the fingertip position and orientation with an accuracy of 1.06±0.62 mm (1.79%±1.05% of length) and 3.58°±2.82° with respect to tendon force and 0.72±0.45 mm (1.22%±0.76% of length) and 2.86°±2.11° with respect to tendon retraction. The retention force of the gripper was predicted with an average error of 0.20±0.12 N.

Keywords: Computer Vision for Automation, Deep Learning for Visual Perception, Wearable Robotics
Abstract: Recently, egocentric action anticipation for wearable robotics cameras has gained considerable attention due to its capability to analyze nouns and verbs from a first-person view. However, this field encounters challenges due to various uncertainties, such as action-irrelevant information and semantically fused representations of verbs and nouns. To overcome these issues, we introduce Ego-A3, designed to improve the robustness and reliability of egocentric action anticipation systems. Ego-A3 adaptively extracts action-relevant data to efficiently utilize additional information beyond visual data. Additionally, Ego-A3 produces effective disentangled representations for verbs and nouns by employing learnable verb and noun queries. Experiments on the EpicKitchens-100 and EGTEA Gaze+ datasets demonstrate that Ego-A3 outperforms existing methods in top-1 accuracy and mean top-5 recall. Our code is publicly available at https://github.com/alsgur0720/egocentric_anticipation.

Keywords: Haptics and Haptic Interfaces, Motion Control, Robust/Adaptive Control
Abstract: This paper proposes a new variable gain sliding mode filter augmented by variable windowing for achieving smooth and reactive response over a broad range of input frequencies. The proposed filter can be seen as a synergistic combination of Kikuuwe et al.’s [1] sliding mode filter with varying gain and sliding surfaces and a novel varying-length moving-window algorithm. In all schemes, the estimated input speed is employed for rendering the filter parameters between low and high settings. The discrete-time algorithm of the proposed filter does not suffer from chattering due to implicit (backward) Euler method. The effectiveness of the proposed filter in achieving better trade-off between noise attenuation and signal preservation is validated in both simulation and experimental scenarios by using the velocity signal obtained by differentiation of quantized position data.

Keywords: Telerobotics and Teleoperation, Physically Assistive Devices
Abstract: The prevailing and most effective approach to teleoperate a robotic arm involves a direct position-to-position mapping, imposing robotic end-effector movements that mirrors those of the user. However, due to this one-to-one mapping, the robot's motions are limited by the user's capability, particularly in translation. Drawing inspiration from head pointers utilized in the 1980s, originally designed to enable drawing with limited head motions for tetraplegic individuals, we proposed a "virtual wand" mapping which could be used by participants with reduced mobility. This mapping employs a virtual rigid linkage between the hand and the robot's end-effector. With this approach, rotations produce amplified translations through a lever arm, creating a "rotation-to-position" coupling and expanding the translation workspace at the expense of a reduced rotation space.

In this study, we compare the virtual wand approach to the one-to-one position mapping through the realization of 6-DoF reaching tasks. Results indicate that the two different mappings perform comparably well, are equally well-received by users, and exhibit similar motor control behaviors. Nevertheless, the virtual wand mapping is anticipated to outperform in tasks characterized by large translations and minimal effector rotations, whereas direct mapping is expected to demonstrate advantages in large rotations with minimal translations. These results pave the way for new interactions and interfaces, particularly in disability assistance utilizing residual body movements (instead of hands) as control input. Leveraging body parts with substantial rotations could enable the accomplishment of tasks previously deemed infeasible with standard direct coupling interfaces.

Keywords: Telerobotics and Teleoperation, Haptics and Haptic Interfaces, Humanoid and Bipedal Locomotion
Abstract: Teleoperated humanoid robot systems have made substantial advancements in recent years, offering a physical avatar that harnesses human skills and decision-making while safeguarding users from hazardous environments. However, current telelocomotion interfaces often fail to accurately represent the robot's environment, limiting the user’s ability to effectively navigate the robot through unstructured terrain. This paper presents an initial telelocomotion framework that integrates the ForceBot locomotion interface with the small-sized humanoid robot, HECTOR V2. The framework utilizes ForceBot to simulate walking motion and estimate the user’s Center of Mass (CoM) trajectory, which serves as a tracking reference for the robot. On the robot side, a model predictive control (MPC) approach, based on a reduced-order single rigid body model, is employed to track the user’s scaled trajectory. We present experimental results on ForceBot’s CoM estimation and the robot’s tracking performance, demonstrating the feasibility of this approach.

Keywords: Telerobotics and Teleoperation, Haptics and Haptic Interfaces, Physical Human-Robot Interaction
Abstract: Variable stiffness of a remote robot is crucial for a teleoperation system to deal with challenging tasks. External stiffness command interfaces have emerged as a promising solution to regulating the remote robot stiffness because of the benefits of their accuracy, ergonomics, and avoidance of the "coupling effect" that usually exists in muscle activity-based stiffness interfaces. However, the use of an external stiffness command interface requires good coordination between two limbs of an operator, which take care of the teleoperation task and the stiffness regulation task, respectively, at the same time, which is demanding for novice operators in dynamic situations necessitating agile and timely stiffness adjustments. In this paper, a new concept of Stiffness Regulation Co-pilot was proposed to facilitate the use of these interfaces. A co-pilot is a virtual agent that consists of a Stiffness Regulation Policy, which infers a reasonable stiffness regulation action from the task performance, and a feedback modality, which conveys the suggested stiffness regulation action to the operator. A preliminary user study was conducted to evaluate the efficacy of the co-pilot and the effect of different modalities of the co-pilot. The results showed that the cutaneous feedback or combined with another modality can potentially improve the task performance of the system and reduce the cognitive load of the operator compared to a teleoperation system without using the co-pilot.

Keywords: Motion Control, Human-Robot Collaboration, Grippers and Other End-Effectors
Abstract: This paper proposes an adaptive neural network synchronous tracking control strategy that can be suitable for event-triggered mechanism in response to the modeling uncertainties and communication delays in bilateral teleoperation systems. Through introducing the event-triggered mechanism with the aim of reducing the network communication frequency in teleoperation system, the master and slave robots communicate with each other only when the triggering conditions are fulfilled, which enhances the efficiency of the network communication. This control strategy can guarantee the exponential convergence of the position synchronization tracking error of the master-slave robot end-effector. Moreover, the event-triggered conditions do not require any empirical design, but can be derived inversely with the aid of the Lyapunov stability theory. And the triggering time interval between two neighboring events is verified to be non-zero. It is demonstrated by utilizing the Lyapunov principle that the presented adaptive neural network control strategy ensures the final asymptotic convergence and exponential convergence of the position synchronization tracking error for master-slave robots under the designed event-triggered mechanisms. Eventually, the feasibility and effectiveness of the developed control strategy are validated by comparative cases.

Keywords: Bimanual Manipulation, Dexterous Manipulation, Learning from Demonstration
Abstract: Aiming to replicate human-like dexterity, perceptual experiences, and motion patterns, we explore learning from human demonstrations using a bimanual system with multifingered hands and visuotactile data. Two significant challenges exist: the lack of an affordable and accessible teleoperation system suitable for a dual-arm setup with multifingered hands, and the scarcity of multifingered hand hardware equipped with touch sensing. To tackle the first challenge, we develop HATO, a low-cost hands-arms teleoperation system that leverages off-the-shelf electronics, complemented with a software suite that enables efficient data collection; the comprehensive software suite also supports multimodal data processing, scalable policy learning, and smooth policy deployment. To tackle the latter challenge, we introduce a novel hardware adaptation by repurposing two prosthetic hands equipped with touch sensors for research. Using visuotactile data collected from our system, we learn skills to complete long-horizon, high-precision tasks which are difficult to achieve without multifingered dexterity and touch feedback. Furthermore, we empirically investigate the effects of dataset size, sensing modality, and visual input preprocessing on policy learning. Our results mark a promising step forward in bimanual multifingered manipulation from visuotactile data. Videos, code, and datasets can be found on: https://toruowo.github.io/hato

Keywords: AI-Based Methods, Bimanual Manipulation, Imitation Learning
Abstract: When performing tasks like laundry, humans naturally coordinate both hands to manipulate objects and anticipate how their actions will change the state of the clothes. However, achieving such coordination in robotics remains challenging due to the need to model object movement, predict future states, and generate precise bimanual actions. In this work, we address these challenges by infusing the predictive nature of human manipulation strategies into robot imitation learning. Specifically, we disentangle task-related state transitions from agent-specific inverse dynamics modeling to enable effective bimanual coordination. Using a demonstration dataset, we train a diffusion model to predict future states given historical observations, envisioning how the scene evolves. Then, we use an inverse dynamics model to compute robot actions that achieve the predicted states. Our key insight is that modeling object movement can help learning policies for bimanual coordination manipulation tasks. Evaluating our framework across diverse simulation and real-world manipulation setups, including multimodal goal configurations, bimanual manipulation, deformable objects, and multi-object setups, we find that it consistently outperforms state-of-the-art state-to-action mapping policies. Our method demonstrates a remarkable capacity to navigate multimodal goal configurations and action distributions, maintain stability across different control modes, and synthesize a broader range of behaviors than those present in the demonstration dataset.

Keywords: Perception for Grasping and Manipulation, Deep Learning in Grasping and Manipulation, Data Sets for Robot Learning
Abstract: Cloth folding is a complex task due to the inevitable self-occlusions of clothes, their complicated dynamics, and the disparate materials, geometries, and textures that garments can have. In this work, we learn folding actions conditioned on text commands. Translating high-level, abstract instructions into precise robotic actions requires sophisticated language understanding and manipulation capabilities. To do that, we leverage a pre-trained vision-language model and repurpose it to predict manipulation actions. Our model, BiFold, can take context into account and achieves stateof-the-art performance on an existing language-conditioned folding benchmark. To address the lack of annotated bimanual folding data, we introduce a novel dataset with automatically parsed actions and language-aligned instructions, enabling better learning of text-conditioned manipulation. BiFold attains the best performance on our dataset and demonstrates strong generalization to new instructions, garments, and environments.

Keywords: Dual Arm Manipulation, Imitation Learning, Visual Servoing
Abstract: We introduce One-Shot Dual-Arm Imitation Learning (ODIL), which enables dual-arm robots to learn precise and coordinated everyday tasks from just a single demonstration of the task. ODIL uses a new three-stage visual servoing (3-VS) method for precise alignment between the end-effector and target object, after which replay of the demonstration trajectory is sufficient to perform the task. This is achieved without requiring prior task or object knowledge, or additional data collection and training following the single demonstration. Furthermore, we propose a new dual-arm coordination paradigm for learning dual-arm tasks from a single demonstration. ODIL was tested on a real-world dual-arm robot, demonstrating state-of-the-art performance across six precise and coordinated tasks in both 4-DoF and 6-DoF settings, and showing robustness in the presence of distractor objects and partial occlusions. Videos are available at https://www.robot-learning.uk/one-shot-dual-arm.

Keywords: Dual Arm Manipulation, Mobile Manipulation, Grasping
Abstract: Picking diverse objects in the real world is a fundamental robotics skill. However, many objects in such settings are bulky, heavy, or irregularly shaped, making them ungraspable by conventional end effectors like suction grippers and parallel jaw grippers (PJGs). In this paper, we expand the range of pickable items without hardware modifications using bimanual nonprehensile manipulation. We focus on a grocery shopping scenario, where a bimanual mobile manipulator equipped with a suction gripper and a PJG is tasked with re- trieving ungraspable items from tightly packed grocery shelves. From visual observations, our method first identifies optimal grasp points based on force closure and friction constraints. If the grasp points are occluded, a series of nonprehensile nudging motions are performed to clear the obstruction. A bimanual grasp utilizing contacts on the side of the end effectors is then executed to grasp the target item. In our replica grocery store, we achieved a 90% success rate over 102 trials in uncluttered scenes, and a 66% success rate over 45 trials in cluttered scenes. We also deployed our system to a real-world grocery store and successfully picked previously unseen items. Our results highlight the potential of bimanual nonprehensile manipulation for in-the-wild robotic picking tasks. A video summarizing this work can be found at youtu.be/g0hOrDuK8jM

Keywords: Bimanual Manipulation, Grasping, Dexterous Manipulation
Abstract: Humans naturally perform bimanual skills to handle large and heavy objects. To enhance a robot's object manipulation capabilities, generating effective bimanual grasp poses is essential. Nevertheless, bimanual grasp synthesis for dexterous hand manipulators remains underexplored. To bridge this gap, we propose the BimanGrasp algorithm for synthesizing bimanual grasps on 3D objects. The BimanGrasp algorithm generates grasp poses by optimizing an energy function that considers grasp stability and feasibility. Furthermore, the quality of the synthesized grasps is verified using the Isaac Gym physics simulation engine. These verified grasp poses form the BimanGrasp-Dataset, which is the first synthesized bimanual dexterous hand grasp pose dataset to our knowledge. The dataset comprises over 150k verified grasps on 900 objects, facilitating the synthesis of bimanual grasps through a data-driven approach. Last, we propose a diffusion model (BimanGrasp-DDPM) trained on the BimanGrasp-Dataset. This model achieved a grasp synthesis success rate of 69.87% and significant acceleration in computational speed compared to BimanGrasp algorithm.

Keywords: Grasping, Perception for Grasping and Manipulation, Cyborgs
Abstract: Achieving a real-time precise grasp of a specified target object in densely cluttered environments is an essential capability for autonomous robot operation. Recently, considerable investigations on planar and spatial grasp have been carried out, and significant results have been obtained. However,these point cloud-based grasp prediction methods often fail to ensure that the generated grasp configurations meet the precise requirements of the task. Additionally, some of the existing grasp pipelines are too time-consuming to meet the demand for real-time robot response. In more challenging cluttered scenes,the quality of pose and gripper jaw opening estimation in highdimensional space requires further improvement. Therefore,this paper introduces a data- and model-independent and efficient method to generate 7-DoF grasp configurations for arbitrary target objects from single-view point cloud data in dense cluttered scenes. In addition, this paper proposes a grasp framework that generates the grasp configuration for the target object while reducing the time consumed during the grasp process, to enable robots to efficiently grasp target objects for designated tasks. The grasp pipeline focuses on guided regions via target detection and rapidly adjusts grasp configurations through multi-region point cloud distribution perception. Extensive real-world robot experiments have demonstrated the effectiveness of the proposed method in grasping target objects in cluttered scenes, achieving higher success rates and reduced runtime compared to baseline methods.The realized code and video are available at https://github.com/L-tj/7DGCG.

Keywords: Grasping
Abstract: In general, humans would grasp an object differently for different tasks, e.g., ``grasping the handle of a knife to cut'' vs. ``grasping the blade to hand over''. In the field of robotic grasp pose detection research, some existing works consider this task-oriented grasping and made some progress, but they are generally constrained by low-DoF gripper type or non-cluttered setting, which is not applicable for human assistance in real life. With an aim to get more general and practical grasp models, in this paper, we investigate a new problem named Task-Oriented 6-DoF Grasp Pose Detection in Clutters (TO6DGC), which extends the task-oriented problem to a more general 6-DOF Grasp Pose Detection in Cluttered (multi-object) scenario. To this end, we construct a large-scale 6-DoF task-oriented grasping dataset, 6-DoF Task Grasp (6DTG), which features 4391 cluttered scenes with over 2 million 6-DoF grasp poses. Each grasp is annotated with a specific task, involving 6 tasks and 198 objects in total. Moreover, we propose One-Stage TaskGrasp (OSTG), a strong baseline to address the TO6DGC problem. Our OSTG adopts a task-oriented point selection strategy to detect where to grasp, and a task-oriented grasp generation module to decide how to grasp given a specific task. To evaluate the effectiveness of OSTG, extensive experiments are conducted on 6DTG. The results show that our method outperforms various baselines on multiple metrics. Real robot experiments also verify that our OSTG has a better perception of the task-oriented grasp points and 6-DoF grasp poses.

Keywords: Grasping, Manipulation Planning, Perception for Grasping and Manipulation
Abstract: Grasping has been a long-standing challenge in facilitating the final interface between a robot and the environment. As environments and tasks become complicated, the need to embed higher intelligence to infer from the surroundings and act on them has become necessary. Although most methods utilize techniques to estimate grasp pose by treating the problem via pure sampling-based approaches in the six-degree-of-freedom space or as a learning problem, they usually fail in real-life settings owing to poor generalization across domains. In addition, the time taken to generate the grasp plan and the lack of repeatability, owing to sampling inefficiency and the probabilistic nature of existing grasp planning approaches, severely limits their application in real-world tasks. This paper presents a lightweight analytical approach towards robotic grasp planning, particularly antipodal grasps, with little to no sampling in the six-degree-of-freedom space. The proposed grasp planning algorithm is formulated as an optimization problem towards estimating grasp points on the object surface instead of directly estimating the end-effector pose. To this extent, a soft-region-growing algorithm is presented for effective plane segmentation, even in the case of curved surfaces. An optimization-based quality metric is then used for evaluation of grasp points to ensure indirect force closure. The proposed grasp framework is compared with existing state-of-the-art grasp planning approach Grasp pose detection (GPD) as baseline over multiple simulated objects. The effectiveness of the proposed approach in comparison to GPD is also evaluated in real-world setting using image and point-cloud data, with the planned grasps being executed using a ROBOTIQ gripper and UR5 manipulator. The proposed approach shows better performance in terms of higher probability for force closure with a complete repeatability.

Keywords: Grasping, Grippers and Other End-Effectors, Manipulation Planning
Abstract: The use of machine learning to investigate grasp affordances has received extensive attention over the past several decades. The existing literature provides a robust basis to build upon, though a number of aspects may be improved. Results commonly work in terms of grasp configuration, with little consideration for the manner in which the grasp may be (re-)produced, from a reachability and trajectory planning perspective. We propose a different perspective on grasp affordance learning, explicitly accounting for grasp synthesis; that is, the manner in which manipulator kinematics are used to allow materialization of grasps. The approach allows to explicitly map the grasp policy space in terms of generated grasp types and associated grasp quality. Results of application to a range of objects illustrate merit of the method and highlight the manner in which it may promote a greater degree of explainability for otherwise intransparent reinforcement processes.

Keywords: Grasping
Abstract: Transparent object grasping remains a persistent challenge in robotics, largely due to the difficulty of acquiring precise 3D information. Conventional optical 3D sensors struggle to capture transparent objects, and machine learning methods are often hindered by their reliance on high-quality datasets. Leveraging NeRF’s capability for continuous spatial opacity modeling, our proposed architecture integrates a NeRF-based approach for reconstructing the 3D information of transparent objects. Despite this, certain portions of the reconstructed 3D information may remain incomplete. To address these deficiencies, we introduce a shape-prior-driven completion mechanism, further refined by a geometric pose estimation method we have developed. This allows us to obtain a complete and reliable 3D information of transparent objects. Utilizing this refined data, we perform scene-level grasp prediction and deploy the results in real-world robotic systems. Experimental validation demonstrates the efficacy of our architecture, showcasing its capability to reliably capture 3D information of various transparent objects in cluttered scenes, and correspondingly, achieve high-quality, stable, and executable grasp predictions.

Keywords: Deep Learning for Visual Perception, Data Sets for Robotic Vision, RGB-D Perception
Abstract: Object grasping is a fundamental challenge in robotics and computer vision, critical for advancing robotic manipulation capabilities. Deformable objects, like fabrics and cloths, pose additional challenges due to their non-rigid nature. In this work, we introduce CeDiRNet-3DoF, a deep-learning model for grasp point detection, with a particular focus on cloth objects. CeDiRNet-3DoF employs center direction regression alongside a localization network, attaining first place in the perception task of ICRA 2023's Cloth Manipulation Challenge. Recognizing the lack of standardized benchmarks in the literature that hinder effective method comparison, we present the ViCoS Towel Dataset. This extensive benchmark dataset comprises 8,000 real and 12,000 synthetic images, serving as a robust resource for training and evaluating contemporary data-driven deep-learning approaches. Extensive evaluation revealed CeDiRNet-3DoF's robustness in real-world performance, outperforming state-of-the-art methods, including the latest transformer-based models. Our work bridges a crucial gap, offering a robust solution and benchmark for cloth grasping in computer vision and robotics.

Keywords: Localization, Probability and Statistical Methods, Robot Safety, Autonomous Vehicle Navigation
Abstract: Deriving safe bounds on particle filter estimate is a research problem that, if solved, could greatly benefit robots in life-critical applications, a field that is facing increasing interest as more robots are being deployed near humans. In response, this paper introduces a new fault detector and derives a performance measure for particle filter: integrity risk. Integrity risk is defined as the probability of having large estimate errors without triggering an alarm, all while considering measurement faults, unknown deterministic errors that cannot be modeled via normal white noise. In this work, the faults come in the form of incorrectly associated features when using the local nearest neighbors. Simulations and experiments assess the efficiency of the introduced safety metric. The results show that safety improves as map density increases as long as the number of particles is sufficient to shape the error distribution and the landmarks are well separated. Also, the results indicate that, when landmarks are poorly separated, particle filter is safer than Kalman filter, whereas, when landmarks are well separated, particle filter is often, but not always, safer than Kalman filter.

Keywords: Localization, Multi-Robot Systems, Wheeled Robots
Abstract: In this paper, we apply lighthouse localization, originally designed for virtual reality motion tracking, to positioning and localization of indoor robots. We first present a lighthouse decoding and tracking algorithm on a low-power wireless microcontroller with hardware implemented in a cm-scale form factor. One-time scene solving is performed on a computer using a variety of standard computer vision tech-niques. Three different robotic localization scenarios are analyzed in this work. The first is a planar scene with a single lighthouse with a four-point pre-calibration. The second is a planar scene with two light-houses that self calibrates with either multiple robots in the experiment or a single robot in motion. The third extends to a 3D scene with two lighthouses and a self-calibration algorithm. The absolute accuracy, measured against a camera-based tracking system, was found to be 7.25 mm RMS for the 2D case and 11.2 mm RMS for the 3D case, respectively. This demonstrates the viability of lighthouse tracking both for small-scale robotics and as an inexpensive and compact alternative to camera-based setups.

Keywords: Localization, Deep Learning for Visual Perception
Abstract: Event cameras are bioinspired sensors with some notable features, including high dynamic range and low latency, which makes them exceptionally suitable for perception in challenging scenarios such as high-speed motion and extreme lighting conditions. In this paper, we explore their potential for localization within pre-existing LiDAR maps, a critical task for applications that require precise navigation and mobile manipulation. Our framework follows a paradigm based on the refinement of an initial pose. Specifically, we first project LiDAR points into 2D space based on a rough initial pose to obtain depth maps, and then employ an optical flow estimation network to align events with LiDAR points in 2D space, followed by camera pose estimation using a PnP solver. To enhance geometric consistency between these two inherently different modalities, we develop a novel frame-based event representation that improves structural clarity. Additionally, given the varying degrees of bias observed in the ground truth poses, we design a module that predicts an auxiliary variable as a regularization term to mitigate the impact of this bias on network convergence. Experimental results on several public datasets demonstrate the effectiveness of our proposed method. To facilitate future research, both the code and the pre-trained models are made available online.

Keywords: Localization, Deep Learning for Visual Perception, Recognition
Abstract: In recent years, robust matching methods using deep learning-based approaches have been actively studied and improved in computer vision tasks. However, there remains a persistent demand for both robust and fast matching techniques. To address this, we propose a novel Mamba-based local feature matching approach, called MambaGlue, where Mamba is an emerging state-of-the-art architecture rapidly gaining recognition for its superior speed in both training and inference, and promising performance compared with Transformer architectures. In particular, we propose two modules: a) MambaAttention mixer to simultaneously and selectively understand the local and global context through the Mamba-based self-attention structure and b) deep confidence score regressor, which is a multi-layer perceptron (MLP)-based architecture that evaluates a score indicating how confidently matching predictions correspond to the ground-truth correspondences. Consequently, our MambaGlue achieves a balance between robustness and efficiency in real-world applications. As verified on various public datasets, we demonstrate that our MambaGlue yields a substantial performance improvement over baseline approaches while maintaining fast inference speed. Our code will be available on https://github.com/url-kaist/MambaGlue.

Keywords: Localization, Range Sensing, Autonomous Vehicle Navigation
Abstract: While UWB-based methods can achieve high localization accuracy in small-scale areas, their accuracy and reliability are significantly challenged in large-scale environments. In this paper, we propose a learning-based framework for Ultra-Wideband (UWB) based localization in such complex large-scale environments, named ULOC. First, anchors are deployed in the environment without knowledge of their actual position. Then, UWB observations are collected when the vehicle travels in the environment. At the same time, map-consistent pose estimates are developed from registering (onboard self-localization) data with the prior map to provide the training labels. We then propose a recurrent neural network (RNN) based on MAMBA that learns the ranging patterns of UWBs over a complex large-scale environment. The experiment demonstrates that our solution can ensure high localization accuracy on a large scale compared to the state-of-the-art. We release our source code to benefit the community at https://github.com/brytsknguyen/uloc.

Keywords: Localization
Abstract: We present a deterministic approach for the localization of an Unmanned Aerial Vehicle (UAV) in a known indoor environment by using only a few downward distance measurements and the corresponding odometries between measurements. For each distance measurement and odometry, we look at the preimage of that distance measurement under the downwards distance function combined with the corresponding odometry where the motion between every two measurements has four degrees of freedom: three of translation and one of azimuth change. The intersection of these preimages yields the set of all possible locations for the UAV. In this work, we present an efficient method for approximating that intersection of preimages. We perform a spatial subdivision search, which splits only voxels containing that intersection. We present a novel technique, based on geometric insights, for correctly evaluating whether a voxel indeed contains a true localization. This technique is also robust under different kinds of errors that might occur. Our method is guaranteed to contain the ground truth location, and its runtime complexity is output sensitive, in the Hausdorff dimension and measure of the resulting intersection of preimages. We demonstrate the effectiveness of this method in various indoor scenarios, showing that it can be used to significantly decrease the uncertainty of localization when solving the kidnapped robot problem in simulation and on a physical drone. Our method can be performed in real-time. Furthermore, our method requires only a map of the environment, odometry and ToF sensors, which is advantageous in terms of cost, privacy and transmission bandwidth. Our open-source software and supplementary materials are available at https://github.com/TAU-CGL/uav-fdml-public.

Keywords: Software Tools for Robot Programming, Software Tools for Benchmarking and Reproducibility
Abstract: Roboticists compare robot motions for tasks such as parameter tuning, troubleshooting, and deciding between possible motions. However, most existing visualization tools are designed for individual motions and lack the features necessary to facilitate robot motion comparison. In this paper, we follow a rigorous design process to create Motion Comparator, a web-based tool that facilitates the comprehension, comparison, and communication of robot motions. Our design process identified roboticists' needs, articulated design challenges, and provided corresponding strategies. Motion Comparator includes several key features such as multi-view coordination, quaternion visualization, time warping, and comparative designs. To demonstrate the applications of Motion Comparator, we discuss four case studies in which our tool is used for motion selection, troubleshooting, parameter tuning, and motion review.

Keywords: Methods and Tools for Robot System Design, Evolutionary Robotics
Abstract: Robot design has traditionally been costly and labor-intensive. Despite advancements in automated processes, it remains challenging to navigate a vast design space while producing physically manufacturable robots. We introduce Text2Robot, a framework that converts user text specifications and performance preferences into physical quadrupedal robots. Within minutes, Text2Robot can use text-to-3D models to provide strong initializations of diverse morphologies. Within a day, our geometric processing algorithms and body-control co-optimization produce a walking robot by explicitly considering real-world electronics and manufacturability. Text2Robot enables rapid prototyping and opens new opportunities for robot design with generative models.

Keywords: Deep Learning Methods, Engineering for Robotic Systems, Software Tools for Robot Programming
Abstract: CAD models are widely used in industry and are essential for robotic automation processes. However, these models are rarely considered in novel AI-based approaches, such as the automatic synthesis of robot programs, as there are no readily available methods that would allow CAD models to be incorporated for the analysis, interpretation, or extraction of information. To address these limitations, we propose QueryCAD, the first system designed for CAD question answering, enabling the extraction of precise information from CAD models using natural language queries. QueryCAD incorporates SegCAD, an open-vocabulary instance segmentation model we developed to identify and select specific parts of the CAD model based on part descriptions. We further propose a CAD question answering benchmark to evaluate QueryCAD and establish a foundation for future research. Lastly, we integrate QueryCAD within an automatic robot program synthesis framework, validating its ability to enhance deep-learning solutions for robotics by enabling them to process CAD models.

Keywords: Software Tools for Robot Programming, Software, Middleware and Programming Environments, Multi-Robot Systems
Abstract: Heterogeneous robots can work together to accomplish a variety of complex tasks and have shown great potential in many fields. There are many efforts to make robot task orchestration more efficient. However, current methods still have some limitations, including the lack of a high-level abstraction for programming method and fault handling mechanism. In this paper, we design a state machine-based, fault-tolerant framework for heterogeneous multi-robot collaboration named HeRo, to effectively support the development of heterogeneous multi-robot systems. HeRo has three key techniques: (1) a state machine-based programming language to flexibly model robot behaviors and tasks; (2) a state synchronization mechanism to achieve information exchange and maintain the consistency among heterogeneous robots in distributed environments; (3) a fault detection and recovery mechanism to monitor the system's runtime states and use Large Language Model (LLM) combined with Planning Domain Definition Language (PDDL) to enable automated recovery. We evaluate the effectiveness and fault recovery capability of the framework by setting up manufacturing task and fault scenarios with varying difficulty in the ARIAC simulation environment, achieving a 100% task completion rate, with low system overhead and flexible scalability.

Keywords: Methods and Tools for Robot System Design, Engineering for Robotic Systems, Optimization and Optimal Control
Abstract: It is challenging to find optimum kinematic designs for non-standard robotic manipulators, e.g., medical, nuclear, and space manipulators, which are demanded to adapt to arbitrary complex tasks in constraints. Such design optimization can be modelled as a multi-dimensional non-convex optimization problem with nonlinear constrained conditions. However, it is non-trivial to ensure the essential reachability condition, i.e., the existence of continuous trajectories between demand positions for serial articulated manipulators, given complex spatial constraints, like obstacles and boundaries. Traditional solutions integrate standard motion planning or inverse kinematics algorithms within a kinematic-design optimization process, resulting in significant demand for time and computing resources. To accelerate design optimization at improved efficiency, we design a novel robust design framework built on a new kinematic design synthesis, which allows for simultaneously optimizing dimension and topology of a serial manipulator's kinematics for arbitrary tasks in constrained environments, using a generalised parametric kinematic model. Significantly, in contrast to standard solutions, we develop a novel computationally effective reachability verification method, which rapidly aborts infeasible motions by exploiting efficient collision checks, based on the Rapidly-exploring Random Tree (RRT) algorithm. The effectiveness of the proposed design framework is verified and evaluated by comparing to baseline benchmarks. Results demonstrate the novel design framework can accelerate kinematic design optimization by an order of magnitude compared to the current state-of-the-art, and optimise link dimension and joint type simultaneously of serial robots for cluttered environments.

Keywords: Embedded Systems for Robotic and Automation, Engineering for Robotic Systems, Methods and Tools for Robot System Design
Abstract: Connected and automated vehicles and robot swarms hold transformative potential for enhancing safety, efficiency, and sustainability in the transportation and manufacturing sectors. Extensive testing and validation of these technologies is crucial for their deployment in the real world. While simulations are essential for initial testing, they often have limitations in capturing the complex dynamics of real-world interactions. This limitation underscores the importance of small-scale testbeds. These testbeds provide a realistic, cost-effective, and controlled environment for testing and validating algorithms, acting as an essential intermediary between simulation and full-scale experiments. This work serves to facilitate researchers' efforts in identifying existing small-scale testbeds suitable for their experiments and provide insights for those who want to build their own. In addition, it delivers a comprehensive survey of the current landscape of these testbeds. We derive 62 characteristics of testbeds based on the well-known sense-plan-act paradigm and offer an online table comparing 23 small-scale testbeds based on these characteristics. The online table is hosted on our designated public webpage https://bassamlab.github.io/testbeds-survey, and we invite testbed creators and developers to contribute to it. We closely examine nine testbeds in this paper, demonstrating how the derived characteristics can be used to present testbeds. Furthermore, we discuss three ongoing ch

Keywords: Transfer Learning, Force and Tactile Sensing, Representation Learning
Abstract: Tactile perception is essential for human interaction with the environment and is becoming increasingly crucial in robotics. Tactile sensors like the BioTac mimic human fingertips and provide detailed interaction data. Despite its utility in applications like slip detection and object identification, this sensor is now deprecated, making many valuable datasets obsolete. However, recreating similar datasets with newer sensor technologies is both tedious and time-consuming. Therefore, adapting these existing datasets for use with new setups and modalities is crucial. In response, we introduce ACROSS, a novel framework for translating data between tactile sensors by exploiting sensor deformation information. We demonstrate the approach by translating BioTac signals into the DIGIT sensor. Our framework consists of first converting the input signals into 3D deformation meshes. We then transition from the 3D deformation mesh of one sensor to the mesh of another, and finally convert the generated 3D deformation mesh into the corresponding output space. We demonstrate our approach to the most challenging problem of going from a low-dimensional tactile representation to a high-dimensional one. In particular, we transfer the tactile signals of a BioTac sensor to DIGIT tactile images. Our approach enables the continued use of valuable datasets and data exchange between groups with different setups.

Keywords: Force and Tactile Sensing, Perception for Grasping and Manipulation, Perception-Action Coupling
Abstract: Manipulating arbitrary objects in unstructured environments is a significant challenge in robotics, primarily due to difficulties in determining an object's center of mass. This paper introduces U-GRAPH: Uncertainty-Guided Rotational Active Perception with Haptics, a novel framework to enhance the center of mass estimation using active perception. Traditional methods often rely on singular interactions and are limited by the inherent inaccuracies of Force-Torque (F/T) sensors. Our approach circumvents these limitations by integrating a Bayesian Neural Network (BNN) to quantify uncertainty and guide the robotic system through multiple, information-rich interactions via grid search and ActiveNet. We demonstrate the remarkable generalizability and transferability of our method with training on a small dataset with limited variation yet still perform well on unseen complex real-world objects.

Keywords: Force and Tactile Sensing, In-Hand Manipulation, Reinforcement Learning
Abstract: Recent progress in reinforcement learning (RL) and tactile sensing has significantly advanced dexterous manipulation. However, these methods often utilize simplified tactile signals due to the gap between tactile simulation and the real world. We introduce a sensor model for tactile skin that enables zero-shot sim-to-real transfer of ternary shear and binary normal forces. Using this model, we develop an RL policy that leverages sliding contact for dexterous in-hand translation. We conduct extensive real-world experiments to assess how tactile sensing facilitates policy adaptation to various unseen object properties and robot hand orientations. We demonstrate that our 3-axis tactile policies consistently outperform baselines that use only shear forces, only normal forces, or only proprioception. Videos and details available on the project website.

Keywords: Representation Learning, Force and Tactile Sensing, Deep Learning in Grasping and Manipulation
Abstract: Tactile sensors differ greatly in design, making it challenging to develop general-purpose methods for processing tactile feedback. In this paper, we introduce a contrastive self-supervised learning approach that represents tactile feedback across different sensor types. Our method utilizes paired tactile data—where two distinct sensors, in our case Soft Bubbles and GelSlims, grasp the same object in the same configuration—to learn a unified latent representation. Unlike current approaches that focus on reconstruction or task-specific supervision, our method employs contrastive learning to create a latent space that captures shared information between sensors. By treating paired tactile signals as positives and unpaired signals as negatives, we show that our model effectively learns a rich, sensor-agnostic representation. Despite significant differences between Soft Bubble and GelSlim sensors, the learned representation enables strong downstream task performance, including zero-shot and few-shot classification and pose estimation. This work provides a scalable solution for integrating tactile data across diverse sensor modalities, advancing the development of generalizable tactile representations.

Keywords: Perception for Grasping and Manipulation, Force and Tactile Sensing
Abstract: An essential problem in robotic systems that are to be deployed in unstructured environments is the accurate and autonomous perception of the shapes of previously unseen objects. Existing methods for shape estimation or reconstruction have leveraged either visual or tactile interactive exploration techniques, or have relied on comprehensive visual or tactile information acquired in an offline manner. In this work, a novel visuo-tactile interactive perception framework - ViTract is introduced for shape estimation of unseen objects. Our framework estimates the shape of diverse objects robustly using low-dimensional, efficient, and generalizable shape primitives, which are superquadrics.

The probabilistic formulation within our framework takes advantage of the complementary information provided by vision and tactile observations while accounting for associated noise. As part of our framework, we propose a novel modality-specific information gain to select the most informative and reliable exploratory action (using vision/tactile) to obtain iterative visuo/tactile information. Our real-robot experiments demonstrate superior and robust performance compared to state-of-the-art visuo-tactile-based shape estimation techniques.

Keywords: Force and Tactile Sensing, Perception for Grasping and Manipulation, Grippers and Other End-Effectors, Barometric Sensing
Abstract: The adoption of tactile sensors in robotics is hindered by their high cost and fragility. We designed and validated a cost-effective and robust barometric tactile sensor array, whose material cost is below 80 USD. Unlike past work, we do not mold the rubber surface over the barometers but instead keep it as a separate element, leading to a design that is easy to fabricate and repair. Machine learning techniques are applied to enhance the sensor’s localization precision, increasing the effective resolution from 6 mm (the distance between adjacent barometers) to 0.284 mm. To investigate the localization model’s robustness, we utilized an E-TRoll robotic gripper to roll differently shaped prismatic objects across the sensing surface mounted on one finger. Under these uncontrolled settings, we achieved a satisfactory real-time localization resolution of within 2.68 mm. Furthermore, we demonstrate a novel practical application: The E-TRoll mimics a 1-DoF parallel gripper inferring a cube’s orientation relative to the sensor. The range of orientations is split into 4 classes, which a trained CNN-LSTM model can predict with an 86.91% five-fold cross-validated accuracy.

Keywords: Human-Centered Automation, Computer Vision for Automation, Continual Learning
Abstract: Robot person following (RPF) is a crucial capability in human-robot interaction (HRI) applications, allowing a robot to persistently follow a designated person. In practical RPF scenarios, the person can often be occluded by other objects or people. Consequently, it is necessary to re-identify the person when he/she reappears within the robot's field of view. Previous person re-identification (ReID) approaches to person following rely on a fixed feature extractor. Such an approach often fails to generalize to different viewpoints and lighting conditions in practical RPF environments. In other words, it suffers from the so-called domain shift problem where it cannot re-identify the person when his re-appearance is out of the domain modeled by the fixed feature extractor. To mitigate this problem, we propose a ReID framework for RPF where we use a feature extractor that is optimized online with both short-term and long-term experiences (i.e., recently and previously observed samples during RPF) using the online continual learning (OCL) framework. The long-term experiences are maintained by a memory manager to enable OCL to update the feature extractor. Our experiments demonstrate that even in the presence of severe appearance changes and distractions from visually similar people, the proposed method can still re-identify the person more accurately than the state-of-the-art methods.

Keywords: Datasets for Human Motion, Wearable Robotics, SLAM
Abstract: Helmet-mounted wearable positioning systems are crucial for enhancing safety and facilitating coordination in industrial, construction, and emergency rescue environments. These systems, including LiDAR-Inertial Odometry (LIO) and Visual-Inertial Odometry (VIO), often face challenges in localization due to adverse environmental conditions such as dust, smoke, and limited visual features. To address these limitations, we propose a novel head-mounted Inertial Measurement Unit (IMU) dataset with ground truth, aimed at advancing data-driven IMU pose estimation. Our dataset captures human head motion patterns using a helmet-mounted system, with data from ten participants performing various activities. We explore the application of neural networks, specifically Long Short-Term Memory (LSTM) and Transformer networks, to correct IMU biases and improve localization accuracy. Additionally, we evaluate the performance of these methods across different IMU data window dimensions, motion patterns, and sensor types. We release a publicly available dataset, demonstrate the feasibility of advanced neural network approaches for helmet-based localization, and provide evaluation metrics to establish a baseline for future studies in this field. Data and code can be found at url{https://lqiutong.github.io/HelmetPoser.github.io/}

Keywords: Human-Robot Collaboration, Cognitive Modeling, Collision Avoidance
Abstract: Human brain possesses the ability to effectively focus on important environmental components, which enhances perception, learning, reasoning, and decision-making. Inspired by this cognitive mechanism, we introduced a novel concept termed relevance for Human-Robot Collaboration (HRC). Relevance is a dimensionality reduction process that incorporates a continuously operating perception module, evaluates cue sufficiency within the scene, and applies a flexible formulation and computation framework. In this paper, we present an enhanced two-loop framework that integrates real-time and asynchronous processing to quantify relevance and leverage it for safer and more efficient human-robot collaboration (HRC). The two-loop framework integrates an asynchronous loop, which leverages an LLM’s world knowledge to quantify relevance, and a real-time loop, which performs scene understanding, human intent prediction, and decision-making based on relevance. HRC decision-making is enhanced by a relevance-based task allocation method, as well as a motion generation and collision avoidance approach that incorporates human trajectory prediction. Simulations and experiments show that our methodology for relevance quantification can accurately and robustly predict the human objective and relevance, with an average accuracy of up to 0.90 for objective prediction and up to 0.96 for relevance prediction. Moreover, our motion generation methodology reduces collision cases by 63.76% and collision frames by 44.74% when compared with a state-of-the-art (SOTA) collision avoidance method. Our framework and methodologies, with relevance, guide the robot on how to best assist humans and generate safer and more efficient actions for HRC.

Keywords: Telerobotics and Teleoperation, Compliance and Impedance Control, Intention Recognition
Abstract: For various teleoperation tasks, position-based control is not practical. An impedance-based control is superior e.g. for handling fragile objects, like harvesting fruits or grasping a paper cup. However, only a few researchers have focused on impedance control for teleoperation. In tele-impedance, the stiffness of a human is measured and transferred to a controller of a robot. Until now, human stiffness was mostly measured either for specific joints or in 2D Cartesian space. We introduce a new way of measuring Cartesian stiffness in 3D using electromyography. Users were asked to perform a peg-in-hole task in three different orientations (0°, 45°, 90°). Meanwhile, electromyography measurements at shoulder and elbow muscle groups are performed. In a proof-of-concept study, we showed that the measured stiffness matrix in Cartesian space differed significantly across the three differently oriented peg-in-hole scenarios. This demonstrates that human stiffness could be predicted in 3D Cartesian space based on the type of task at hand.

Keywords: Gesture, Posture and Facial Expressions, Human Detection and Tracking, Omnidirectional Vision
Abstract: Fisheye cameras offer robots the ability to capture human movements across a wider field of view (FOV) than standard pinhole cameras, making them particularly useful for applications in human-robot interaction and automotive contexts. However, accurately detecting human poses in fisheye images is challenging due to the curved distortions inherent to fisheye optics. While various methods for undistorting fisheye images have been proposed, their effectiveness and limitations for poses that cover a wide FOV has not been systematically evaluated in the context of absolute human pose estimation from monocular fisheye images. To address this gap, we evaluate the impact of pinhole, equidistant and double sphere camera models, as well as cylindrical projection methods, on 3D human pose estimation accuracy. We find that in close-up scenarios, pinhole projection is inadequate, and the optimal projection method varies with the FOV covered by the human pose. The usage of advanced fisheye models like the double sphere model significantly enhances 3D human pose estimation accuracy. We propose a heuristic for selecting the appropriate projection model based on the detection bounding box to enhance prediction quality. Additionally, we introduce and evaluate on our novel FISHnCHIPS dataset, which features 3D human skeleton annotations in fisheye images, including images from unconventional angles, such as extreme close-ups, ground-mounted cameras, and wide-FOV poses.

Keywords: Modeling and Simulating Humans, Human and Humanoid Motion Analysis and Synthesis, Intention Recognition
Abstract: Predicting human motion may lead to considerable advantages for human-robot interaction, particularly when precise synchronization between the robot’s motion and the user’s movement is imperative. The inherent stochastic nature of human behavior, combined with the restricted window of response, can give rise to residual and undesirable forces during interactions, potentially harming the user. Therefore, efficient prediction of human joint movements may enhance the performance of various interaction control frameworks used in wearable robots. This paper proposes the HuMAn algorithm for predicting human joint motion based on a recurrent neural network. This algorithm consists of a long-term memory network, used to interpret sequences of poses, and a prediction layer, employed to build the most likely future user poses within a specified time horizon. Network training was performed using datasets encompassing various subjects and types of motion. The results demonstrate the effectiveness of the proposed algorithm, as evidenced by average general prediction errors below 0.1 radians for predictive horizons of up to 500 milliseconds. Furthermore, a mean absolute error of 0.026 radians was achieved for a periodic treadmill walk. Simulation results demonstrate a large improvement in transparency control performance in a case study with an upper limb exoskeleton robot.

Keywords: Perception-Action Coupling, Representation Learning
Abstract: Vision language models have played a key role in extracting meaningful features for various robotic applications. Among these, Contrastive Language-Image Pretraining (CLIP) is widely used in robotic tasks that require both vision and natural language understanding. However, CLIP was trained solely on static images paired with text prompts and has not yet been fully adapted for robotic tasks involving dynamic actions. In this paper, we introduce Robotic-CLIP to enhance robotic perception capabilities. We first gather and label large-scale action data, and then build our Robotic-CLIP by fine-tuning CLIP on 309,433 videos (~7.4 million frames) of action data using contrastive learning. By leveraging action data, Robotic-CLIP inherits CLIP's strong image performance while gaining the ability to understand actions in robotic contexts. Intensive experiments show that our Robotic-CLIP outperforms other CLIP-based models across various language-driven robotic tasks. Additionally, we demonstrate the practical effectiveness of Robotic-CLIP in real-world grasping applications.

Keywords: Learning from Demonstration, Imitation Learning, Data Sets for Robot Learning
Abstract: In-context imitation learning is the capability to perform novel tasks when prompted with task demonstration examples. In-Context Robot Transformer (ICRT) is a causal transformer that performs autoregressive prediction on sensorimotor trajectories, which include images, proprioceptive states, and actions. This approach supports flexible and trainingfree execution of new tasks at test time. Experiments with a Franka Emika robot demonstrate that ICRT can adapt to new environment configurations that differ from both the prompt and the training data. In a multi-task environment setup, ICRT significantly outperforms current state-of-the-art robot foundation models on generalization to unseen tasks. Code, data, and appendix are available on https://icrt.dev.

Keywords: Incremental Learning, Deep Learning for Visual Perception, Planning under Uncertainty
Abstract: Current methods based on Neural Radiance Fields fail in the low data limit, particularly when training on incomplete scene data. Prior works augment training data only in next-best-view applications, which lead to hallucinations and model collapse with sparse data. In contrast, we propose adding a set of views during training by rejection sampling from a posterior uncertainty distribution, generated by combining a volumetric uncertainty estimator with spatial coverage. We validate our results on partially observed scenes; on average, our method performs 39.9% better with 87.5% less variability across established scene reconstruction benchmarks, as compared to state of the art baselines. We further demonstrate that augmenting the training set by sampling from any distribution leads to better, more consistent scene reconstruction in sparse environments. This work is foundational for robotic tasks where augmenting a dataset with informative data is critical in resource-constrained, a priori unknown environments. Videos and source code are available at https://murpheylab.github.io/low-data-nerf.

Keywords: Imitation Learning, Learning from Demonstration, Representation Learning
Abstract: In this paper, we leverage self-supervised vision transformer models and their emergent semantic abilities to improve the generalization abilities of imitation learning policies. We introduce DVK, an imitation learning algorithm that leverages rich pre-trained Visual Transformer patch-level embeddings to obtain better generalization when learning through demonstrations. Our learner sees the world by clustering appearance features into groups associated with semantic concepts, forming stable keypoints that generalize across a wide range of appearance variations and object types. We demonstrate how this representation enables generalized behaviour by evaluating imitation learning across a diverse dataset of object manipulation tasks. To facilitate further study of generalization in Imitation Learning, all of our code for the method and evaluation, as well as the dataset, is made available.

Keywords: Big Data in Robotics and Automation, Sensorimotor Learning, Learning from Demonstration
Abstract: Interacting with the world is a multi-sensory experience: achieving effective general-purpose interaction requires making use of all available modalities -- including vision, touch, and audio -- to fill in gaps from partial observation. For example, when vision is occluded reaching into a bag, a robot should rely on its senses of touch and sound. However, state-of-the-art generalist robot policies are typically trained on large datasets to predict robot actions solely from visual and proprioceptive observations. In this work, we propose FuSe, a novel approach that enables finetuning visuomotor generalist policies on heterogeneous sensor modalities for which large datasets are not readily available by leveraging natural language as a common cross-modal grounding. We combine a multimodal contrastive loss with a sensory-grounded language generation loss to encode high-level semantics. In the context of robot manipulation, we show that FuSe enables performing challenging tasks that require reasoning jointly over modalities such as vision, touch, and sound in a zero-shot setting, such as multimodal prompting, compositional cross-modal prompting, and descriptions of objects it interacts with. We show that the same recipe is applicable to widely different generalist policies, including both diffusion-based generalist policies and large vision-language-action (VLA) models. Extensive experiments in the real world show that FuSe is able to increase success rates by over 20% compared to all considered baselines.

Keywords: Deep Learning for Visual Perception, Perception for Grasping and Manipulation, Deep Learning Methods
Abstract: Humans have the remarkable ability to use held objects as tools to interact with their environment. For this to occur, humans internally estimate how hand movements affect the object's movement. We wish to endow robots with this capability. We contribute methodology to jointly estimate the geometry and pose of objects grasped by a robot, from RGB images captured by an external camera. Notably, our method transforms the estimated geometry into the robot's coordinate frame, while not requiring the extrinsic parameters of the external camera to be calibrated. Our approach leverages 3D foundation models, large models pre-trained on huge datasets for 3D vision tasks, to produce initial estimates of the in-hand object. These initial estimations do not have physically correct scales and are in the camera's frame. Then, we formulate, and efficiently solve, a coordinate-alignment problem to recover accurate scales, along with a transformation of the objects to the coordinate frame of the robot. Forward kinematics mappings can subsequently be defined from the manipulator's joint angles to specified points on the object. These mappings enable the estimation of points on the held object at arbitrary configurations, enabling robot motion to be designed with respect to coordinates on the grasped objects. We empirically evaluate our approach on a robot manipulator holding a diverse set of real-world objects.

Keywords: Machine Learning for Robot Control, Model Learning for Control, Compliance and Impedance Control
Abstract: This paper proposes a novel force-based task-orientation controller for interaction tasks with environmental orientation uncertainties. The main aim of the controller is to align the robot tool along the main task direction (e.g., along screwing, insertion, polishing, etc.) without the use of any external sensors (e.g., vision systems), relying only on end-effector wrench measurements/estimations. We propose a gradient descent-based orientation controller, enhancing its performance with the orientation predictions provided by a Gaussian Process model. Derivation of the controller is presented, together with simulation results (considering a probing task) and experimental results involving various re-orientation scenarios, i.e., i) a task with the robot in interaction with a soft environment, ii) a task with the robot in interaction with a stiff and inclined environment, and iii) a task to enable the assembly of a gear into its shaft. The proposed controller is compared against a state-of-the-art approach, highlighting its ability to re-orient the robot tool even in complex tasks (where the state-of-the-art method fails).

Keywords: Reinforcement Learning, Imitation Learning, Machine Learning for Robot Control
Abstract: This paper proposes a data-driven model-free inverse reinforcement learning (IRL) algorithm tailored for solving an inverse H_infty control problem. In the problem, both an expert and a learner engage in H_infty control to reject disturbances and the learner's objective is to imitate the expert's behavior by reconstructing the expert's performance function through IRL techniques. Introducing zero-sum game principles, we first formulate a model-based single-loop IRL policy iteration algorithm that includes three key steps: updating the policy, action, and performance function using a new correction formula and the standard inverse optimal control principles. Building upon the model-based approach, we propose a model-free single-loop off-policy IRL algorithm that eliminates the need for initial stabilizing policies and prior knowledge of the dynamics of expert and learner. Also, we provide rigorous proof of convergence, stability, and Nash optimality to guarantee the effectiveness and reliability of the proposed algorithms. Furthermore, we showcase the efficiency of our algorithm through simulations and experiments, highlighting its advantages compared to the existing methods.

Keywords: Machine Learning for Robot Control, Sensorimotor Learning, Learning from Demonstration
Abstract: Differentiable simulation has become a powerful tool for system identification. While prior work has focused on identifying robot properties using robot-specific data or object properties using object-specific data, our approach calibrates object properties by using information from the robot, without relying on data from the object itself. Specifically, we utilize robot joint encoder information, which is commonly available in standard robotic systems. Our key observation is that by analyzing the robot's reactions to manipulated objects, we can infer properties of those objects, such as inertia and softness. Leveraging this insight, we develop differentiable simulations of robot-object interactions to inversely identify the properties of the manipulated objects. Our approach relies solely on proprioception — the robot’s internal sensing capabilities — and does not require external measurement tools or vision-based tracking systems. This general method is applicable to any articulated robot and requires only joint position information. We demonstrate the effectiveness of our method on a low-cost robotic platform, achieving accurate mass and elastic modulus estimations of manipulated objects with just a few seconds of computation on a laptop.

Keywords: Machine Learning for Robot Control, Deep Learning Methods, Reinforcement Learning
Abstract: Adapting reinforcement learning (RL) policies to various robot body configurations is a significant challenge for creating flexible autonomous systems. This study presents a novel framework that integrates Reservoir Computing (RC) with the First-Order Reduced and Controlled Error (FORCE) learning rule to enhance policy adaptability in RL. The RC serves as a dynamic feature extractor, capturing temporal dependencies by pre-training on state transitions generated through random actions. This pre-training acts as regularization, reducing variance and preventing overfitting to specific configurations Subsequently, the control policy network is trained on a limited set of body variations using the enriched features from the RC. Experimental results across three distinct environments demonstrate that the proposed RC+FORCE framework significantly improves policy performance and adaptability to unseen robot configurations compared to traditional reinforcement learning through domain randomization. These findings highlight the effectiveness of combining RC-based feature extraction with FORCE-based training in developing robust RL agents.

Keywords: Machine Learning for Robot Control, Aerial Systems: Mechanics and Control, Robust/Adaptive Control
Abstract: A critical goal of adaptive control is enabling robots to rapidly adapt in dynamic environments. Recent studies have developed a meta-learning-based adaptive control scheme, which uses meta-learning to extract nonlinear features (represented by Deep Neural Networks (DNNs)) from offline data, and uses adaptive control to update linear coefficients online. However, such a scheme is fundamentally limited by the linear parameterization of uncertainties and does not fully unleash the capability of DNNs. This paper introduces a novel learning-based adaptive control framework that pretrains a DNN via self-supervised meta-learning (SSML) from offline trajectories and online adapts the full DNN via composite adaptation. In particular, the offline SSML stage leverages the time consistency in trajectory data to train the DNN to predict future disturbances from history, in a self-supervised manner without environment condition labels. The online stage carefully designs a control law and an adaptation law to update the full DNN with stability guarantees. Empirically, the proposed framework significantly outperforms (19-39%) various classic and learning-based adaptive control baselines, in challenging real-world quadrotor tracking problems under large dynamic wind disturbance.

Keywords: Machine Learning for Robot Control, Vision-Based Navigation, Legged Robots
Abstract: First-order Policy Gradient (FoPG) algorithms such as Backpropagation through Time and Analytical Policy Gradients leverage local simulation physics to accelerate policy search, significantly improving sample efficiency in robot control compared to standard model-free reinforcement learning. However, FoPG algorithms can exhibit poor learning dynamics in contact-rich tasks like locomotion. Previous approaches address this issue by alleviating contact dynamics via algorithmic or simulation innovations. In contrast, we propose guiding the policy search by learning a residual over a simple baseline policy. For quadruped locomotion, we find that the role of residual policy learning in FoPG-based training (FoPG RPL) is primarily to improve asymptotic rewards, compared to improving sample efficiency for model-free RL. Additionally, we provide insights on applying FoPG's to pixel-based local navigation, training a point-mass robot to convergence within seconds. Finally, we showcase the versatility of FoPG RPL by using it to train locomotion and perceptive navigation end-to-end on a quadruped in minutes.

Keywords: Imitation Learning, Integrated Planning and Learning, Sensor Fusion
Abstract: Multimodal end-to-end autonomous driving has shown promising advancements in recent work. By embedding more modalities into end-to-end networks, the system’s understanding of both static and dynamic aspects of the driving environment is enhanced, thereby improving the safety of autonomous driving. In this paper, we introduce METDrive, an end-to-end system that leverages temporal guidance from the embedded time series features of ego states, including rotation angles, steering, throttle signals, and waypoint vectors. The geometric features derived from perception sensor data and the time series features of ego state data jointly guide the waypoint prediction with the proposed temporal guidance loss function. We evaluated METDrive on the CARLA leaderboard benchmarks, achieving a driving score of 70%, a route completion score of 94%, and an infraction score of 0.78.

Keywords: Learning from Demonstration, Representation Learning, Imitation Learning
Abstract: Large real-world driving datasets have sparked significant research into various aspects of learning-based motion planners for autonomous driving. These include data augmentation, model architecture, reward design, training strategies, and planner pipelines. In this paper, we review and benchmark previous methods. Experiments show that many of these approaches have limited generalization abilities in planning performance due to overly complex designs or training paradigms. Experiments further reveal that as models are appropriately scaled, many designs become redundant. Therefore, we introduce StateTransformer-2 (STR2), a scalable, decoder-only motion planner. STR2uses a Vision Transformer (ViT) encoder and a mix-of-experts (MoE) causal transformer architecture. The MoE backbone addresses modality collapse and reward balancing by expert routing during training. Extensive experiments on the NuPlan dataset show that our method generalizes better than previous approaches across different test sets and closed-loop simulations. We evaluate its scalability on billions of real-world urban driving scenarios, demonstrating consistent accuracy improvements as both data and model size grow.

Keywords: Computer Vision for Automation, Autonomous Vehicle Navigation
Abstract: All-weather image restoration (AWIR) is crucial for reliable autonomous navigation under adverse weather conditions. AWIR models are trained to address a specific set of weather conditions such as fog, rain, and snow. But this causes them to often struggle with out-of-distribution (OoD) samples or unseen degradations which limits their effectiveness for real-world autonomous navigation. To overcome this issue, existing models must either be retrained or fine-tuned, both of which are inefficient and impractical, with retraining needing access to large datasets, and fine-tuning involving many parameters. In this paper, we propose using Low-Rank Adaptation (LoRA) to efficiently adapt a pre-trained all-weather model to novel weather restoration tasks. Furthermore, we observe that LoRA lowers the performance of the adapted model on the pre-trained restoration tasks. To address this issue, we introduce a LoRA-based fine-tuning method called LoRA-Align (LoRA-A) which seeks to align the singular vectors of the fine-tuned and pre-trained weight matrices using Singular Value Decomposition (SVD). This alignment helps preserve the model's knowledge of its original tasks while adapting it to unseen tasks. We show that images restored with LoRA and LoRA-A can be effectively used for computer vision tasks in autonomous navigation, such as semantic segmentation and depth estimation. Project page: https://sudraj2002.github.io/loraapage/.

Keywords: Computer Vision for Automation, AI-Based Methods, Vision-Based Navigation
Abstract: 3D detection of vehicles is an essential component for autonomous driving applications. Nevertheless, collecting the supervised training data for learning 3D vehicle detectors would be costly (e.g. utilization of expensive LiDAR sensors) and labor-intensive (for human annotation). In comparison to 3D detection, 2D object detection has achieved a well-developed status, boosting stable and robust performance with widespread application in numerous fields, thanks to the large scale (i.e. amount of samples) of existing training datasets of 2D object detection. Hence, in our work, we propose to realize 3D detection via leveraging the robustness of 2D detectors and developing a network that lifts 2D detections to 3D. With the flexibility of building upon various backbone models (e.g. the models which take image regions detected by 2D detector as inputs to predict their corresponding 3D bounding boxes, or the existing monocular 3D detection models which have the intermediate output of 2D bounding boxes), we propose several geometry-driven objectives, including projection consistency loss, geometry depth loss, and opposite bin loss, to improve the training upon 2D-to-3D lifting. Our extensive experimental results demonstrate that our proposed geometry-driven objectives not only contribute to the superior results of 3D detection but also provide better generalizability across datasets.

Keywords: Autonomous Vehicle Navigation, Simulation and Animation, AI-Based Methods
Abstract: We present LidarDM, a novel LiDAR generative model capable of producing realistic, layout-aware, physically plausible, and temporally coherent LiDAR videos. LidarDM stands out with two unprecedented capabilities in LiDAR generative modeling: (i) LiDAR generation guided by driving scenarios, offering significant potential for autonomous driving simulations, and (ii) 4D LiDAR point cloud generation, enabling the creation of realistic and temporally coherent sequences. At the heart of our model is a novel integrated 4D world generation framework. Specifically, we employ latent diffusion models to generate the 3D scene, combine it with dynamic actors to form the underlying 4D world, and subsequently produce realistic sensory observations within this virtual environment. Our experiments indicate that our approach outperforms competing algorithms in realism, temporal coherency, and layout consistency. We additionally show that LidarDM can be used as a generative world model simulator for training and testing perception models.

Keywords: Computer Vision for Automation, Planning, Scheduling and Coordination, Object Detection, Segmentation and Categorization
Abstract: End-to-end autonomous driving with vision-only is not only more cost-effective compared to LiDAR-vision fusion but also more reliable than traditional methods. To achieve a economical and robust purely visual autonomous driving system, we propose RenderWorld, a vision-only end-to-end autonomous driving framework, which generates 3D occupancy labels using a self-supervised gaussian-based Img2Occ Module, then encodes the labels by AM-VAE, and use world model for forecasting and planning. RenderWorld employs Gaussian Splatting to represent 3D scenes and render 2D images greatly improves segmentation accuracy and reduces GPU memory consumption compared with NeRF-based methods. By applying AM-VAE to encode air and non-air separately, RenderWorld achieves more fine-grained scene element representation, lead- ing to state-of-the-art performance in both 4D occupancy forecasting and motion planning from autoregressive world model.

Keywords: Actuation and Joint Mechanisms, Mechanism Design, Legged Robots
Abstract: Repetitive subtasks of locomotion are offloaded from a conventional computer-actuator-sensor set-up to automatic mechanical processes. The subtasks considered are: (1) when out-of-contact with the environment, move a leg to a ready position in preparation for contact, and (2) when contact is detected, push off the ground. Using conventional closed loop control, subtask (1) would be accomplished by programming logic and a feedback loop onto a computer- motor-encoder system, and subtask (2) would be accomplished by sensing contact, then commanding the leg motor to push- off via programmed computer logic. We demonstrate how to transition this programmed logic from a computer processor to a mechanical processor. The mechanical processor as an example of mechanical intelligence performs preprogrammed actions based on combinations of components states, some of which are internal and some that interact with the environment. Because signals are not digital, but rather mechanical quantities of energy, position, and force; transitioning to a mechanical processor enables a third subtask not possible by the computer alone: that is, (3) the accumulation of elastic energy while out-of-contact with the environment, and its automatic release upon contact for a more powerful push-off motion. Migrating processing out of the computer allows for faster responses to dynamic events, and instantiates a high-powered reflex triggered by ground contact.

Keywords: Wearable Robotics, Physically Assistive Devices, Human Performance Augmentation
Abstract: We present an end-to-end control framework for powered prostheses using temporal convolutional networks (TCNs) to map onboard sensor inputs directly to torque and position outputs. Unlike traditional hierarchical controllers that require extensive manual tuning, our approach offers a user-independent, deep learning-based solution for dynamic locomotion. Using data from 17 participants across five walking modes (i.e. level-ground, ramps, stairs), we trained two TCN models, validated via leave-one-subject-out cross-validation. Our models achieved torque RMSEs of 0.126 Nm/kg at the ankle and 0.348 Nm/kg at the knee, and position RMSEs of 2.87° and 5.64°, respectively. Corresponding R² values were 0.56 (ankle) and 0.54 (knee) for the torque model, and 0.66 (ankle) and 0.87 (knee) for the position model. The results suggest that TCNs can replicate impedance-based control outputs, taking a step closer to real-world deployability of powered prostheses.

Keywords: Mechanism Design, Grippers and Other End-Effectors, Legged Robots
Abstract: Manipulation with legs on multi-pedal robots is not a new concept, but it has rarely been considered on wheel-legged robots due to their mechanical structure. This paper presents a mechanism that transitions between a wheel and a gripper, allowing wheel-legged robots to perform manipulation tasks with their legs. The mechanism uses the same motor for both wheeled locomotion and gripping, and no new actuator is needed for transitioning. It is capable of storing small objects, liquid, or soil while still functioning as a wheel. Our analysis showed that those distinctive modes corresponded to the factors of the complex kinematics polynomials of the mechanism. The two modes exist because the polynomials that govern the mechanism's kinematics factor. That is to say, its overall motion is the union of two lower degree irreducible components. A single-leg platform was built to demonstrate its ability to grasp objects and collect water or soil samples, and explore its capability as a contact force sensor through experiments. Such a design could enable wheel-legged robots to perform more diverse loco-manipulation tasks without installing additional limbs.

Keywords: Collision Avoidance, Constrained Motion Planning
Abstract: Artificial potential fields are used for safety in operational space control by generating repulsive forces near obstacles. They have significant limitations, like causing oscillations near obstacles or at high speeds, affecting dynamics away from unsafe areas, and needing extensive tuning to prevent local minima. Our proposed approach, AIKIDO, aims to maximise motion accuracy and minimise tuning effort, according to the principle that the collision scene geometry, plus the definition of a clearance value, naturally encodes in maximal coordinates safe and unsafe configurations and control inputs. Our key insight is that we can leverage iterative dynamics to test control inputs before applying them to the real system. In our method, the clearance value acts as an exact configurable (even at runtime) safety margin, which defines exactly the minimum allowable distance between collision shapes (e.g., a link and an obstacle). In a receding horizon fashion, we can iteratively predict near-future feasible states adhering closely to control inputs, up to necessary corrections (distance constraints). Through simulation experiments, we demonstrate unmatched motion accuracy in the proximity of obstacles and self-collisions, which enables our method to exhibit superior performance in challenging, general-purpose, arm positioning tasks.

Keywords: Cognitive Control Architectures, Embodied Cognitive Science, Multi-Robot Systems
Abstract: This poster introduces HARMONIC, a cognitive-robotic architecture that integrates the LEIA cognitive framework with general-purpose robot control systems applied to human-robot teaming (HRT). We also present a cognitive strategy for robots that incorporates metacognition, natural language communication, and explainability capabilities required for collaborative partnerships in HRT. Through our preliminary simulation experiments involving a joint search task performed by a heterogeneous team of a UGV, a drone, and a human operator, we demonstrate the system's ability to coordinate actions between robots with heterogeneous capabilities, adapt to complex scenarios, and facilitate natural human-robot communication. Evaluation results show that robots using the LEIA architecture within the HARMONIC framework can reason about plans, goals, and team member attitudes while providing clear explanations for their decisions, which are essential prerequisites for realistic human-robot teaming.

Keywords: Telerobotics and Teleoperation, Grasping
Abstract: Sharing control with an autonomous controller is a promising approach to reduce the operator's workload and enhance performance in the execution of complex tasks. To assist the operator, the autonomous controller must predict their intention, by monitoring biological signals. In natural manipulation, eye movements play an anticipatory role, making gaze a meaningful signal for decoding human intent. However, the role of gaze in teleoperation remains unclear. We investigated this by analyzing gaze behavior during a teleoperated task, finding that participants directed their gaze toward task-relevant areas, with fixations also on the hand or the moving object. Once the user’s intention is inferred, the robot should assist by autonomously executing actions such as grasping. To this end, we propose a human-inspired reasoning algorithm that reasons about the optimal grasping strategy. The agreement between the decisions of our reasoning algorithm and human behavior in grasping was assessed through a questionnaire in which participants expressed their preferences on how to grasp objects. A significant alignment was observed across most tested conditions. The observed misalignments highlight the importance of balancing the principles that guide human grasping with the needs of robots. In conclusion, by combining the prediction of the human intention with a human-inspired reasoning algorithm for grasp selection, we aim to enhance the collaboration between human operators and robots.

Keywords: Legged Robots, Reinforcement Learning
Abstract: Reinforcement learning (RL)-based robot control has emerged as a promising approach. However, the transition between simulation and real situations remains a critical challenge, especially for robots equipped with high-speed actuators. To address this issue, we integrate an ideal BLDC motor model into the simulation and introduce Time Delay Neural Network (TDNN) that accumulates multiple observations over time. The motor modeling allows us to bridge the gap between simulation and reality, and the TDNN allows the policy to implicitly learn the internal nonlinearities of the actuators. In addition, we introduce an estimator network that simultaneously predicts the robot’s terrain height, foot-ground contact state, and body linear velocity. By combining this estimator with an asymmetric actor-critic architecture and concurrently training it, the trained policy demonstrates that the robot can climb stairs of 17 cm in height and descend stairs of up to 27.5 cm, demonstrating an effective transition from simulation to real situations.

Keywords: Optimization and Optimal Control, Motion and Path Planning, Whole-Body Motion Planning and Control
Abstract: The generation of optimal trajectories for high-dimensional robotic systems under constraints remains computationally challenging due to the need to simultaneously satisfy dynamic feasibility, input limits, and task-specific objectives while searching over high-dimensional spaces. Recent approaches using the Affine Geometric Heat Flow (AGHF) Partial Differential Equation (PDE) have demonstrated promising results, generating dynamically feasible trajectories for complex systems like the Digit V3 humanoid within seconds. These methods efficiently solve trajectory optimization problems over a two-dimensional domain by evolving an initial trajectory to minimize control effort. However, these AGHF approaches are limited to a single type of optimal control problem (i.e., minimizing the integral of squared control norms) and typically require initial guesses that satisfy constraints to ensure satisfactory convergence. These limitations restrict the potential utility of the AGHF PDE especially when trying to synthesize trajectories for robotic systems. This paper generalizes the AGHF formulation to accommodate arbitrary cost functions, significantly expanding the classes of trajectories that can be generated. This work also introduces a Phase 1 - Phase 2 Algorithm that enables the use of constraint-violating initial guesses while guaranteeing satisfactory convergence. The effectiveness of the proposed method is demonstrated through comparative evaluations.

Keywords: Nonholonomic Motion Planning, Motion and Path Planning, Aerial Systems: Mechanics and Control
Abstract: A kinematic (turn-radius-constrained) vehicle captures the relevant dynamics for global path planning around obstacle fields for many applications. More advanced trajectory optimizers can also use these paths as their initial guess. However, state-of-the-art heuristic graph planners ignore U-turn arcs which may result from the turn radius constraint of the vehicle and sampling-based graphs optimize the path very slowly. These heuristic graphs rapidly find the globally optimal path in cases where the vehicle’s turn radius is large compared to its environment.

Keywords: Wearable Robotics, Prosthetics and Exoskeletons, Mechanism Design
Abstract: Powered exoskeletons aim to improve human mobility by assisting the user’s biological limbs with robotic actuators. However, conventional robotic actuators with a fixed transmission ratio suffer from an inherent trade-off between motor size, output inertia, and output torque. Therefore, powered exoskeletons utilizing conventional actuators are typically heavy and loud with high joint impedance or lack sufficient torque and power to meaningfully assist the user in challenging activities. These drawbacks significantly diminish the net benefit of wearing powered exoskeletons, limiting their clinical success.

In this poster, we present a powered knee exoskeleton that combines a direct-ball screw drive with a variable transmission in a novel torque-sensitive actuator. Benchtop experiments confirm the actuator’s high torque density, low impedance, and quiet operation. Preliminary able-bodied human subject testing verifies the ability of the exoskeleton to provide a significant portion of the user’s biological torque and power in multiple activities. By enabling lightweight, quiet, and powerful exoskeletons, the proposed actuation concept has the potential to improve the clinical benefits of powered knee exoskeletons.

Keywords: Underactuated Robots, Nonholonomic Mechanisms and Systems, Model Learning for Control
Abstract: Geometric mechanics provides valuable insights into how biological and robotic systems use changes in shape to move by mechanically interacting with their environment. This perspective produced an approach for obtaining simplified data-driven models for locomotion systems directly from motion tracking data. Here we focus on the locomotion of under-actuated robotic systems with passive shape degrees of freedom (DoF), interacting with a granular fluid - a regime we intentionally selected because it is hard to model. We compared four modeling approaches, predicting body velocity both within gait and across gaits. Switching our model from a phase dependent linear model to a manifold learning approach reduced velocity prediction error by 55% but required 6 times as much data; including the passive DoF in the models reduced prediction errors by 4%. These improvements compound, and yield overall R^2 values above 90%, demonstrating that our data-driven geometric mechanics models for locomotion systems can produce highly predictive models even where no first principles equations of motion are known.

Keywords: Assembly, Computer Vision for Automation, Object Detection, Segmentation and Categorization
Abstract: NIST leads and supports multiple efforts to develop standards for 3D imaging systems used for industrial automation through ASTM subcommittee E57.23 on Industrial 3D Machine Vision Systems. One of these standards aims to measure the performance of robotic bin-picking systems and NIST has established a testbed with six different bin-picking systems that use a variety of technologies. A working group led by industry experts, is formulating the metrics, artifacts, and test methods to evaluate bin-picking systems. NIST is supporting this effort by performing tests, analyzing data, and developing artifacts. The effort has proposed definitions for terms such as bin-picking vision system, robot cycle time, vision cycle time (VCT), total cycle time (TCT), and bin-picking throughput. NIST conducted tests to understand the effects on a bin-picking system’s throughput of task parameters such as part count, bin depth, part position in bin, part location in FoV, and part geometry/complexity, while keeping vision system parameters constant. Preliminary results based on a representative 3D imaging system show that VCT increases when part number > 10, is not correlated to bin depth or part location, and has slight correlation with robot position and access within a bin. Results also showed that TCT and throughput are strongly correlated with part complexity/symmetry.

Keywords: Prosthetics and Exoskeletons, Wearable Robotics, Mechanism Design
Abstract: Workplace injuries cost the United States over 160 billion dollars annually. Of these workplace injuries, approximately 29% are musculoskeletal disorders (MSDs). Previous research has shown that both passive and active exoskeletons can reduce muscle activation for a wide variety of tasks, increase lifting endurance, and decrease the risk of MSDs. Passive exoskeletons tend to be low-profile but have less ability to adapt to multiple movements and use cases, whereas powered exoskeletons can be controlled to benefit multiple movements at the expense of larger size. Here, we propose the design of a low-profile and high torque-density, powered hip and back support exoskeleton to assist with ergonomics in a range of workplace tasks. By assisting the lower back and hips, we aim to reduce muscle effort in lifting tasks, reduce the risk of MSD, and reduce the exertion required for ambulation. According to dynamic simulations, our proposed actuator achieves torques up to 34 Nm of hip torque, representing approximately 35% of the biological torque required for squatting/lifting movement for an average US male. The actuator can also provide up to 31%, 87%, and 35% of the torque required for walking, stair ascent, and stair descent, respectively. The actuator is compatible with exoskeleton interfaces from the on-market IX Back Air (SuitX, USA), which allows for easy donning and doffing. The device adds just 2.8 cm laterally to the body and does not interfere with sitting.

Keywords: Force and Tactile Sensing, Compliant Joints and Mechanisms, Grippers and Other End-Effectors
Abstract: A capacitive-type force/torque sensor (FTS) is particularly advantageous for compact and miniature designs. However, miniaturizing the sensor introduces several challenges, such as imbalanced sensitivity that leads to increased coupling errors and a reduced electrode area, which degrades sensing performance and measurement sensitivity. Typically, a six-axis FTS requires at least six electrode components, each dedicated to independently detecting force and torque along normal and shear directions. However, distinguishing and placing electrodes for normal and shear detection becomes spatially inefficient and impractical in miniaturized designs. To address these limitations, we propose a structurally decoupled compliant structure. This structure enables independent measurement along four axes—three for force and one for torque. Furthermore, we present a simplified capacitive sensing approach that does not require the separation of electrodes by direction. Instead, flat electrodes are attached directly to the compliant structure. By isotropically arranging these sensing components, an assembled compact sensing unit capable of six-axis force-torque measurement is realized. This paper presents the proposed capacitive sensing unit's design concept and simulation results.

Keywords: Prosthetics and Exoskeletons, Wearable Robotics, Rehabilitation Robotics
Abstract: Powered ankle exoskeletons have the potential to improve mobility by assisting the ankle joint during ambulation. To achieve this goal, ankle exoskeletons must provide substantial assistive torque in a small and lightweight form. Unfortunately, most exoskeletons are too heavy and bulky for practical application. They often lack protective covers and require wearing a backpack with wires running down the user’s legs, lessening their robustness and utility in the real world. This poster presents a lightweight and compact powered ankle exoskeleton with fully integrated and enclosed actuation, batteries, and electronics. The proposed design features a linear series-elastic actuator (SEA), achieving high torque/power density while enabling high-fidelity closed-loop torque control. The proposed ankle exoskeleton was investigated using a variable stiffness impedance controller with three healthy subjects walking on a treadmill. The results of benchtop experiments show a -3 dB bandwidth exceeding 19 Hz. The results of experiments with healthy subjects show a torque density of 37 Nm/kg and a power density of 81.5 W/kg. Experiments with one stroke subject demonstrate that exoskeleton assistance results in more normative ankle and knee kinematics and increases foot clearance. This compact and lightweight ankle exoskeleton design satisfies practical requirements for real-world use while providing sufficient torque density to improve mobility.

Keywords: Force and Tactile Sensing, Force Control, Modeling, Control, and Learning for Soft Robots
Abstract: Deformable Object Manipulation (DOM) remains a critical challenge in robotics due to the complexities of developing suitable model-based control strategies. Deformable Tool Manipulation (DTM) further complicates this task by introducing additional uncertainties between the robot and its environment. While humans effortlessly manipulate deformable tools using touch and experience, robotic systems struggle to maintain stability and precision. To address these challenges, we present a novel State-Adaptive Koopman LQR (SA-KLQR) control framework for real-time deformable tool manipulation, demonstrated through a case study in environmental swab sampling for food safety. This method leverages Koopman operator-based control to linearize nonlinear dynamics while adapting to state-dependent variations in tool deformation and contact forces. A tactile-based feedback system dynamically estimates and regulates the swab tool’s angle, contact pressure, and surface coverage, ensuring compliance with food safety standards. Additionally, a sensor-embedded contact pad monitors force distribution to mitigate tool pivoting and deformation, improving stability during dynamic interactions. Experimental results validate the SA-KLQR approach, demonstrating accurate contact angle estimation, robust trajectory tracking, and reliable force regulation. The proposed framework bridging the gap between data-driven learning and optimal control in robotic interaction tasks.

Keywords: Soft Robot Materials and Design, Soft Robot Applications
Abstract: Soft robots can achieve exceptional adaptability through tunable morphological and mechanical properties. Incorporating materials with dynamically adjustable characteristics can enhance this versatility further. Granular metamaterials, consisting of discrete particles with individually variable properties, offer a promising approach to bulk property adaptation by adjusting the properties of constituent particles. This work introduces variable size and variable stiffness (VS²) particles, in which both particle size and stiffness are independently modulated through concentric pneumatic chambers. We characterize the achievable workspace, mapping particle responses to independent chamber inflation. To demonstrate their use in a granular assembly, we arrange an array of VS² particles in a 2D hexagonal packing and validate that behavior in packed configurations aligns with free-space characterizations. This study establishes a foundation for adaptive granular materials and provides a platform for further computational and experimental exploration of granular metamaterials with tunable properties.

Keywords: Mapping, Probabilistic Inference, Deep Learning for Visual Perception
Abstract: This paper introduces a novel probabilistic mapping algorithm, LatentBKI, which enables open-vocabulary mapping with quantifiable uncertainty. Traditionally, semantic mapping algorithms focus on a fixed set of semantic categories which limits their applicability for complex robotic tasks. Vision-Language (VL) models have recently emerged as a technique to jointly model language and visual features in a latent space, enabling semantic recognition beyond a predefined, fixed set of semantic classes. LatentBKI recurrently incorporates neural embeddings from VL models into a voxel map with quantifiable uncertainty, leveraging the spatial correlations of nearby observations through Bayesian Kernel Inference (BKI). LatentBKI is evaluated against similar explicit semantic mapping and VL mapping frameworks on the popular Matterport3D and Semantic KITTI datasets, demonstrating that LatentBKI maintains the probabilistic benefits of continuous mapping with the additional benefit of open-dictionary queries. Real-world experiments demonstrate applicability to challenging indoor environments.

Keywords: Robot Safety, Collision Avoidance, Nonholonomic Motion Planning
Abstract: The safe navigation of nonholonomic robots in environments filled with dynamic obstacles requires control strategies that not only guarantee collision-free trajectories but also adapt to real-time changes. In this work, we introduce a novel Control Barrier Function candidate-the Dynamic Parabolic Control Barrier Function (DPCBF)-designed for the kinematic bicycle model. Unlike traditional approaches such as the collision cone-based CBF (C3BF), which relies on a fixed geometric safety boundary and can become overly conservative under high obstacle density, the DPCBF dynamically adjusts the safe set. By redefining safety in a rotated coordinate frame where the x-axis aligns with the relative position vector between the robot and an obstacle, the DPCBF employs a parabolic function whose curvature and translation are tuned according to the current relative distance and velocity. Extensive simulation studies demonstrate that the DPCBF outperforms the conventional C3BF by maintaining feasibility within the quadratic programming (QP) formulation under scenarios with both static and multiple moving obstacles. The adaptive nature of the parabolic safe set allows the robot to respond effectively to varying collision risks, ensuring fewer QP infeasibilities and more reliable navigation towards the goal.

Keywords: Human-Centered Robotics, Human Factors and Human-in-the-Loop, Design and Human Factors
Abstract: As assistive robots increasingly support Activities of Daily Living (ADLs), real-time Cognitive Fatigue (CF) assessment is crucial for safety and performance in Human-Robot Interaction (HRI). Existing approaches often lack multimodal sensing, continuous estimation, or adaptive mechanisms. This study presents a novel framework for automatic CF assessment using data from wearable physiological sensors, such as Electrocardiography (ECG) and Electrodermal Activity (EDA), collected from 16 participants while performing collaborative ADL tasks with a mobile manipulator. CF was induced via N-back tasks and validated using the Visual Analogue Scale for Fatigue (VAS-F). Machine learning models, including Random Forest Regressor (RFR), Support Vector Regression (SVR), and Gradient Boosting Machine (GBM) were applied using the epoch-based pipeline. RFR achieved the highest accuracy in predicting the VAS-F score using only ECG data (R² = 0.89, RMSE = 4.13) and by combining ECG and EDA data (R² = 0.89, RMSE = 4.85). These findings demonstrate the feasibility of continuous CF estimation with wearable sensors. This offline modeling serves as a foundation for integrating real-time adaptation in assistive robotics, enabling personalized and safer HRI.

Keywords: Marine Robotics, Virtual Reality and Interfaces, Human-Robot Collaboration
Abstract: Working underwater presents many challenges for human divers; however, the assistance of an underwater robot has the potential to reduce risks by collaborating with a diver and reducing the workload. This work presents a virtual reality (VR) application that simulates a BlueROV2, an underwater robot, to implement and evaluate an underwater robot-to-human interaction scenario, with the aim of improving the effectiveness of communication underwater while collaborating on an inspection and manipulation task. The use of VR allows for more repeatable assessments of robotic performance, with lower risks associated to human users, while providing a level of immersion in the task that compares to humans being underwater wearing goggles. By developing an immersive VR game, combined with Wizard-of-Oz techniques which allow for structured scenarios where the robot can be triggered to interrupt at varying points in the collaboration task, the effects of attention-grabbing strategies and the choices of signal set on human performance can be assessed, as well as the robot’s ability to redirect a human collaborator in underwater tasks.

Keywords: Aerial Systems: Applications, Dual Arm Manipulation, Aerial Systems: Mechanics and Control
Abstract: This study presents a drone-mounted dual-arm shovel system designed for efficient object retrieval in complex and remote environments. Addressing key challenges—such as object detection, spatial localization, precision landing, adaptive gripping, and stable airborne transport—the system integrates a rotating camera, servo-driven shovels, and an onboard computer running a deep learning-based detection model. Through synchronized motion and iterative multi-angle analysis, the system accurately identifies, assesses, and retrieves target objects. Experimental validation demonstrates reliable performance in detecting, grasping, weighing, and transporting various items. This work lays the foundation for advanced aerial manipulation, expanding the potential of drones in remote search, recovery, and logistics applications.

Keywords: Humanoid Robot Systems, Multifingered Hands, Grippers and Other End-Effectors
Abstract: Robotic hands inspired by human anatomy are being developed in various forms to perform diverse manipulation tasks. However, faithfully replicating the human hand—comprising complex structures and numerous degrees of freedom (DoFs)—leads to increased manufacturing cost and complexity. In practice, not all DoFs are necessary for every task, and simplified designs often omit or couple certain joints to enhance efficiency. Designing such hands requires a careful balance between task capability and mechanical simplicity, which in turn calls for a quantitative framework to guide DoF configuration based on task requirements.

In this study, we propose a method for quantitatively analyzing the task performance of robotic hands with reduced DoFs. By introducing a joint-task matrix that captures the contribution of individual joints to specific tasks, and deriving a joint coupling matrix to reveal dependencies between joints, we provide a systematic approach to evaluate hand configurations. This enables decision-making based on quantitative evaluation when designing simplified robotic hands tailored to specific functional needs.

Keywords: Legged Robots, Passive Walking, Humanoid and Bipedal Locomotion
Abstract: The stability of bipedal system during walking or running depends on both the sagittal and frontal plane dynamics. In the frontal plane, the stability is affected by the hip offset, to the extent that adjusting stride time can lead to stable oscillations without active control, which can reduce the control effort and energy expenditure. However, there is limited understanding of how key parameters—such as mass, stiffness, leg length, and hip width—affect stability and the minimum stride frequency needed to maintain it. This study aims to address these gaps through analying how individual model parameters and the system's natural frequency influence the minimum stride frequency required for stable locomotion. Stability is evaluated through eigenvalue analysis of the linearized Poincaré map. We propose a method to predict the minimum stride frequency for various models, and compare predicted stride frequencies with actual values for randomly generated models, showing a maximum error of approximately 1%.

Keywords: Multi-Robot Systems, Task and Motion Planning, Planning, Scheduling and Coordination
Abstract: We introduce Lazy-DaSH, an improvement over the recent state of the art multi-robot task and motion planning method DaSH, which scales to more than twice the number of robots and objects while achieving an order of magnitude faster planning when applied to a multi-manipulator object rearrangement problem. We achieve this improvement through a hierarchical approach, where a high-level task planning layer identifies planning spaces required for task completion, and motion feasibility is validated lazily only within these spaces. In contrast, DaSH precomputes the motion feasibility of all possible actions, resulting in higher costs for constructing state space representations. Lazy-DaSH ensures efficient query performance by utilizing a hierarchical constraint feedback mechanism, effectively conveying motion feasibility to the query process. By maintaining smaller state space representations, our method significantly reduces both representation construction time and query time. We evaluate Lazy-DaSH in two scenarios, demonstrating its scalability with increasing numbers of robots and objects, as well as its adaptability in resolving conflicts through the constraint feedback.

Keywords: Legged Robots, Formal Methods in Robotics and Automation
Abstract: We propose an integrated planning framework for quadrupedal locomotion over dynamically changing, unforeseen terrains. Existing approaches either rely on heuristics for instantaneous foothold selection--compromising safety and versatility--or solve expensive trajectory optimization problems with complex terrain features and long time horizons. In contrast, our framework leverages reactive synthesis to generate correct-by-construction controllers at the symbolic level, and mixed-integer convex programming (MICP) for dynamic and physically feasible footstep planning for each symbolic transition. We use a high-level manager to reduce the large state space in synthesis by incorporating local environment information, improving synthesis scalability. To handle specifications that cannot be met due to dynamic infeasibility, and to minimize costly MICP solves, we leverage a symbolic repair process to generate only necessary symbolic transitions. During online execution, re-running the MICP with real-world terrain data, along with runtime symbolic repair, bridges the gap between offline synthesis and online execution. We demonstrate, in simulation, our framework's capabilities to discover missing locomotion skills and react promptly in safety-critical environments, such as scattered stepping stones and rebars.

Keywords: Autonomous Agents, Agent-Based Systems, Semantic Scene Understanding
Abstract: Recent progress in using Large Language Models (LLMs) for embodied task planning has yielded impressive results in unconstrained environments. However, their inherent unreliability—particularly due to hallucinated or inconsistent action predictions—remains a significant challenge. In contrast, formal planning methods offer strong reliability guarantees through explicit logical representations of the environment, but struggle to scale to the complexity and ambiguity of open-world tasks. This work explores the complementary strengths of these paradigms. We introduce the Formally Augmented Language Agent (FALA), a hybrid task planning framework that defers to formal methods when possible for robust decision-making, and invokes LLM reasoning only when symbolic representations fall short. This principled integration yields both high reliability and flexibility. We evaluate FALA extensively on the ALFRED benchmark for embodied task planning, achieving a 99% task completion rate using GPT-4o—substantially outperforming prior methods in both accuracy and robustness. Through ablation studies, we demonstrate that the addition of FALA's Precondition Verification and Domain Independent Planner to a zero-shot LLM agent significantly improves overall planning performance by an average of ~40% and ~60%, respectively, across 7 different LLM backbones on the ALFRED benchmark. We believe this work represents a major step towards more efficient and reliable LLM planning

Keywords: Constrained Motion Planning, Motion and Path Planning, Machine Learning for Robot Control
Abstract: Constrained motion planning requires robots to satisfy certain conditions, such as keeping a cup upright, while completing a task. Solving for connected regions in the configuration space that satisfy these constraints can be very difficult as the functions used to model these constraints can be highly nonlinear and computationally expensive. Current state-of-the-art methods build piecewise-linear approximations of constraint manifolds to model valid configurations, and additional steps are taken to ensure connectivity between neighboring linear approximations. However, these numerical continuation methods have several drawbacks: they are numerically sensitive, they are incapable of handling inequality constraints, and they have several parameters that are difficult to properly tune. To solve these problems, the proposed method uses a neural network-based approach to learn constraint functions and compute continuous representations of constraint manifolds. The neural network is first decomposed into many different linear output regions. Then, each of these regions is checked to determine whether the constraint function can be satisfied within the given region. The proposed method finally produces a set of polytopes that represent valid configurations for the given constraint.

Keywords: Space Robotics and Automation, Cellular and Modular Robots, Robotics and Automation in Construction
Abstract: The author’s research group has been conducting an advanced project under Japan's Moonshot R&D Program, aiming to develop a heterogeneous, collaborative multi-robot system for lunar resource exploration and outpost construction. The system features modular robots with reconfigurable mechanical components, enabling adaptive functions and efficient reuse. Plug-and-play AI controllers are being developed to allow each robot to autonomously recognize its configuration and coordinate tasks based on situational context. To validate the concept, field tests using “MoonBots” were conducted at a lunar analog site on JAXA’s Sagamihara campus. Although composed of relatively simple modules—namely “Wheel” and “Arm” units—MoonBots demonstrated diverse capabilities such as self-assembly, rough terrain traversal, material transport, and structural deployment, through modular reconfiguration. Development of the AI system is ongoing, with a focus on achieving real-time coordination, fault tolerance, and dynamic task allocation. Pre-deployment simulations and sim-to-real transfer techniques have accelerated progress and improved reliability. Results confirm the feasibility and scalability of modular, intelligent robot teams for future lunar base construction. The up-to-date results are presented in the late breaking poster session.

Keywords: Automation at Micro-Nano Scales, Micro/Nano Robots, Nanomanufacturing
Abstract: Noncontact manipulation of soft micro-fibers has great potential in advanced manufacturing, materials science, and biomedical engineering. However, current noncontact manipulation techniques primarily focus on objects with regular shapes, e.g., solid particles, cells, or droplets, with fewer solutions available for manipulating flexible and elongated structures. In this paper, an automated ultrasonic manipulation system is introduced for in-plane soft micro-fiber manipulation, which mainly consists of an ultrasonic transducer array and a microscope. A real-time trap generation algorithm is designed to manipulate the micro-fibers by the visual feedback from microscope. An adequate theoretical analysis is also provided for explanation of the deformation behavior of micro-fiber under external forces. The system is capable of precise in-plane positioning and motion trajectory planning to micro-fiber end, and in-plane morphological reshaping to the micro-fiber. Experiments validated the effectiveness of the proposed system for the in-plane manipulation of soft micro-fibers. Finally, the system was showcased by the practical application of material property characterization.

Keywords: Motion and Path Planning, Constrained Motion Planning, Optimization and Optimal Control
Abstract: The moving target traveling salesman problem with obstacles (MT-TSP-O) seeks an obstacle-free trajectory for an agent that intercepts a given set of moving targets, each within a specified time windows, and returns to the agent's starting position. Each target moves with a constant velocity within its time windows, and the agent has a speed limit no smaller than any target's speed. We present FMC*-TSP, the first complete and bounded-suboptimal algorithm for the MT-TSP-O, and results for an agent whose configuration space is in R^3. Our algorithm interleaves a high-level search and a low-level search where the high-level search solves a generalized traveling salesman problem with time windows (GTSP-TW) to find a sequence of targets and corresponding time windows for the agent to visit. Given such a sequence, the low-level search then finds an associated agent trajectory. To solve the low-level planning problem, we develop a new algorithm called FMC*, which finds a shortest path on a graph of convex sets (GCS) via implicit graph search and pruning techniques specialized for problems with moving targets. We test FMC*-TSP on 280 problem instances with up to 40 targets and demonstrate its smaller median runtime than a baseline based on prior work.

Keywords: Task Planning, Reinforcement Learning
Abstract: The efficient planning of stacking boxes, especially in the online setting where the sequence of item arrivals is unpredictable, remains a critical challenge in modern warehouse and logistics management. Existing solutions often address box size variations, but overlook their intrinsic and physical properties, such as density and rigidity, which are crucial for real-world applications. We use reinforcement learning (RL) to solve this problem by employing action space masking to direct the RL policy towards valid actions. Unlike previous methods that rely on heuristic stability assessments which are difficult to assess in physical scenarios, our framework utilizes online learning to dynamically train the action space mask, eliminating the need for manual heuristic design. Extensive experiments demonstrate that our proposed method outperforms existing state-of-the-arts. Furthermore, we deploy our learned task planner in a real-world robotic palletizer, validating its practical applicability in operational settings.