Keywords: Humanoid and Bipedal Locomotion
Abstract: This work explores an innovative algorithm designed to enhance the mobility of underactuated bipedal robots across challenging terrains, especially when navigating through spaces with constrained opportunities for foot support, like steps or stairs. By combining ankle torque with a refined angular momentum-based linear inverted pendulum model (ALIP), our method allows variability in the robot's center of mass height. We employ a dual-strategy controller that merges virtual constraints for precise motion regulation across essential degrees of freedom with an ALIP-centric model predictive control (MPC) framework, aimed at enforcing gait stability. The effectiveness of our feedback design is demonstrated through its application on the Cassie bipedal robot, which features 20 degrees of freedom. Key to our implementation is the development of tailored nominal trajectories and an optimized MPC that reduces the execution time to under 500 microseconds—and, hence, is compatible with Cassie's controller update frequency. This paper not only showcases the successful hardware deployment but also demonstrates a new capability, a bipedal robot using a moving walkway.

Keywords: Humanoid and Bipedal Locomotion, Legged Robots, Modeling, Control, and Learning for Soft Robots
Abstract: Flexibility and softness play a significant role in dynamic human motions. This includes the flexibility owing to ligaments in the human body and the softness of external structures such as a leaf-spring-type prosthesis. Thus, robotic systems need to utilize such flexibility to achieve dynamic and energy-efficient motion. In this study, we proposed a compliance optimization-based control framework for a rigid-soft hybrid robot system where the continuous deformation of a flexible structure is represented using the piece-wise constant strain (PCS) model. We divided the hybrid system into two states: single support and double support. We validated the proposed method in these states using forward dynamics simulations, assuming a hybrid link system that consists of a humanoid robot with a flexible prosthesis.

Keywords: Humanoid and Bipedal Locomotion, Humanoid Robot Systems, Imitation Learning
Abstract: Recently, humanoid robots have made significant advances in their ability to perform complex tasks due to the deployment of Reinforcement Learning (RL), however, the inherent complexity of humanoid robots, including the difficulty of planning complex reward functions and training entire complex systems, still poses a notable challenge. To conquer these challenges, after many iterations and in-depth investigations, we have meticulously developed a full-size humanoid robot, "Adam", whose innovative structural design greatly improves the efficiency and effectiveness of the imitation learning process. In addition, we have developed a novel imitation learning framework based on an adversarial motion prior, which applies not only to Adam but also to humanoid robots in general. Using the framework, Adam can exhibit unprecedented human-like characteristics in locomotion tasks. Our experimental results demonstrate that the proposed framework enables Adam to achieve human-comparable performance in complex locomotion tasks, marking the first time that human locomotion data has been used for imitation learning in a full-size humanoid robot.

Keywords: Humanoid and Bipedal Locomotion, Reinforcement Learning, Motion Control
Abstract: Accurate state estimation plays a critical role in ensuring the robust control of humanoid robots, particularly in the context of learning-based control policies for legged robots. However, there is a notable gap in analytical research concerning estimations. Therefore, we endeavor to further understand how various types of estimations influence the decision-making processes of policies. In this paper, we provide quantitative insight into the effectiveness of learned state estimations, employing saliency analysis to identify key estimation variables and optimize their combination for humanoid locomotion tasks. Evaluations assessing tracking precision and robustness are conducted on comparative groups of policies with varying estimation combinations in both simulated and real-world environments. Results validated that the proposed policy is capable of crossing the sim-to-real gap and demonstrating superior performance relative to alternative policy configurations.

Keywords: Humanoid and Bipedal Locomotion, Whole-Body Motion Planning and Control, Legged Robots
Abstract: We propose an effective whole-body MPC controller for locomotion of humanoid robots. Our method generates motions using the full kinematics, allowing it to account for joint limits and to exploit upper-body motions to reject disturbances. Each MPC iteration solves a single QP that considers the interplay between dynamic and kinematic features of the robot. Thanks to our special formulation, we are able to perform a feasibility analysis, which opens the door to future enhancements of functionality and performance, e.g., step adaptation in complex environments. We demonstrate its effectiveness through a campaign of dynamic simulations aimed at highlighting how the joint limits and the use of the angular momentum through upper-body motions are fundamental for maximizing performance, robustness, and ultimately make the robot able to execute more challenging gaits.

Keywords: Humanoid and Bipedal Locomotion, Reinforcement Learning, Machine Learning for Robot Control
Abstract: In this work, we introduce a control framework that combines model-based footstep planning with Reinforcement Learning (RL), leveraging desired footstep patterns derived from the Linear Inverted Pendulum (LIP) dynamics. Utilizing the LIP model, our method forward predicts robot states and determines the desired foot placement given the velocity commands. We then train an RL policy to track the foot placements without following the full reference motions derived from the LIP model. This partial guidance from the physics model allows the RL policy to integrate the predictive capabilities of the physics-informed dynamics and the adaptability characteristics of the RL controller without overfitting the policy to the template model. Our approach is validated on the MIT Humanoid, demonstrating that our policy can achieve stable yet dynamic locomotion for walking and turning. We further validate the adaptability and generalizability of our policy by extending the locomotion task to unseen, uneven terrain. During the hardware deployment, we have achieved forward walking speeds of up to 1.5 m/s on a treadmill and have successfully performed dynamic locomotion maneuvers such as 90-degree and 180-degree turns.

Keywords: Humanoid and Bipedal Locomotion, Robust/Adaptive Control, Reinforcement Learning
Abstract: A necessary capability for humanoid robots is the ability to stand and walk while rejecting natural disturbances. Recent progress has been made using sim-to-real reinforcement learning (RL) to train such locomotion controllers, with approaches differing mainly in their reward functions. However, prior works lack a clear method to systematically test new reward functions and compare controller performance through repeatable experiments. This limits our understanding of the trade-offs between approaches and hinders progress. To address this, we propose a low-cost, quantitative benchmarking method to evaluate and compare the real-world performance of standing and walking (SaW) controllers on metrics like command following, disturbance recovery, and energy efficiency. We also revisit reward function design and construct a minimally constraining reward function to train SaW controllers. We experimentally verify that our benchmarking framework can identify areas for improvement, which can be systematically addressed to enhance the policies. We also compare our new controller to state-of-the-art controllers on the Digit humanoid robot. The results provide clear quantitative trade-offs among the controllers and suggest directions for future improvements to the reward functions and expansion of the benchmarks.

Keywords: Humanoid and Bipedal Locomotion, Motion and Path Planning, Robot Safety
Abstract: We study the problem of bipedal robot navigation in complex environments with uncertain and rough terrain. In particular, we consider a scenario in which the robot is expected to reach a desired goal location by traversing an environment with uncertain terrain elevation. Such terrain uncertainties induce not only untraversable regions but also robot motion perturbations. Thus, the problems of terrain mapping and locomotion stability are intertwined. We evaluate three different kernels for Gaussian process (GP) regression to learn the terrain elevation. We also learn the motion deviation resulting from both the terrain as well as the discrepancy between the reduced-order Prismatic Inverted Pendulum Model used for planning and the full-order locomotion dynamics. We propose a hierarchical locomotion-dynamics-aware sampling-based navigation planner. The global navigation planner plans a series of local waypoints to reach the desired goal locations while respecting locomotion stability constraints. Then, a local navigation planner is used to generate a sequence of dynamically feasible footsteps to reach local waypoints. We develop a novel trajectory evaluation metric to minimize motion deviation and maximize information gain of the terrain elevation map. We evaluate the efficacy of our planning framework on Digit bipedal robot simulation in MuJoCo.

Keywords: Humanoid and Bipedal Locomotion, Legged Robots, Wheeled Robots
Abstract: Wheel-legged robots combine the advantages of both wheeled robots and legged robots, offering versatile locomotion capabilities with excellent stability on challenging terrains and high efficiency on flat surfaces. However, existing wheel-legged robots typically have limited hip joint mobility compared to humans, which play a crucial role in locomotion. In this paper, we introduce Whleaper, a novel 10-degree-of-freedom (DOF) bipedal wheeled robot, with 3 DOFs at the hip of each leg. Its humanoid joint design enables adaptable motion in complex scenarios, ensuring stability and flexibility. This paper introduces the details of Whleaper, with a focus on innovative mechanical design, control algorithms and system implementation. Firstly, stability stems from the increased DOFs at the hip, which maximize motion range, enhance the attitude of contact between the feet and the ground. Secondly, the extra DOFs also augment its mobility capabilities. During walking or sliding, more complex movements can be adopted to execute obstacle avoidance tasks. Thirdly, we utilize two control algorithms to implement multimodal motion for walking and sliding. By controlling specific DOFs of the robot, we conducted a series of simulation and practical experiments, demonstrating that a high-DOF hip joint design can effectively enhance the stability and flexibility of wheel-legged robots. Whleaper shows its capability to perform actions such as squatting, obstacle avoidance sliding, and rapid turning in real-world scenarios.

Keywords: Humanoid Robot Systems, Humanoid and Bipedal Locomotion, Model Learning for Control
Abstract: The ability to accomplish manipulation and locomotion tasks in the presence of significant time-varying external loads is a remarkable skill of humans that has yet to be replicated convincingly by humanoid robots. Such an ability will be a key requirement in the environments we envision deploying our robots: dull, dirty, and dangerous. External loads constitute a large model bias, which is typically unaccounted for. In this work, we enable our humanoid robot to engage in loco-manipulation tasks in the presence of significant model bias due to external loads. We propose an online estimation and control framework involving the combination of a physically consistent extended Kalman filter for inertial parameter estimation coupled to a whole-body controller. We showcase our results both in simulation and in hardware, where weights are mounted on Nadia's wrist links as a proxy for engaging in tasks where large external loads are applied to the robot.

Keywords: Humanoid Robot Systems, Vision-Based Navigation, Humanoid and Bipedal Locomotion
Abstract: Feasible regions for continuous walking must provide necessary information for footstep planning, including surrounding landing areas and details about obstacles to be avoided during foot swing. However, the current frame lacks sufficient information to construct a feasible region needed at the current moment due to knee occlusion. To this end, this paper uses polygon merging to construct an information-complete feasible region. This polygon merging refers to merging polygons from the current frame and a specific previous frame. Since the polygon is more concise and efficient than point cloud for environmental representation, construction can be completed quickly without GPU acceleration. Experiments show that the proposed method successfully constructs informative feasible regions within the allowed time frame, enabling the robot to navigate stairs.

Keywords: Humanoid Robot Systems, Force and Tactile Sensing, Sensor Networks
Abstract: For humanoid whole body contact motions, it is important to recognize the existence of whole body contacts and the contact forces. The challenges in recognizing the existence of whole body contacts and the contact forces in life-size humanoids are: 1) the measurement part with low mechanical strength must be tolerant of high load and 2) it is difficult to model thick elastic bodies with high impact tolerance and uneven sensor placements when applied to various shapes of the whole body. This paper proposes a method of constructing a load tolerant tactile sensor by separating the loaded part from the measuring part with magnetism and protecting the measuring part inside the frame of the robot. For modeling difficulties, this paper proposes learning the relationship between the change in the detected physical quantity due to deformation of the elastic body and the contact force distribution. This paper shows through experiments that the proposed tactile sensor based on a robot frame is load tolerant enough to support the weight of a life-sized humanoid, and that it can acquire contact force distribution and the robot is able to acclimate to external forces.

Keywords: Humanoid Robot Systems, Task Planning, Formal Methods in Robotics and Automation
Abstract: A pilot needs to manipulate various gadgets in the cockpit based on vast knowledge of rules and procedures while verbally communicating with air traffic controllers. While precision manipulation in the cockpit during the flight is already a difficult task, a far more difficult thing is how to make a robot learn all the knowledge needed to fly an airplane in accordance with all the rules and regulations. As a pioneering effort, this paper introduces LLM-PIBOT, which leverages the latest advances in Large Language Models (LLMs) to empower a humanoid pilot robot (PIBOT) to take the full authority of an airplane by understanding and executing complex procedures outlined in Pilot's Operating Handbooks (POHs). Unlike traditional rule-based methods, LLM-PIBOT system infers suitable flight procedures, employs an embedding process to accurately identify relevant procedures within documents, and structures the text-extracted flight tasks into tuples using our carefully crafted prompts. This approach enables PIBOT to adapt to the given POHs, generating and executing task plans in real-time in response to commands and situations. Experimental results show that LLM-PIBOT can comprehend and follow the complex procedures specified in the manuals and to fly the airplane on a full-scale simulator using the generated flight plans.

Keywords: Humanoid Robot Systems, Education Robotics, Product Design, Development and Prototyping
Abstract: Humanoid robots have potential applications across diverse sectors, including education, healthcare, and customer service. This paper presents a project on designing and building a low-cost humanoid robot equipped with a flex sensor- based movement mechanism, highlighting its compatibility with Raspberry Pi and microcontrollers such as Arduino Uno and Nano. The project aims to investigate the robot's relevance and effectiveness within educational settings to showcase how a low- cost humanoid robot can potentially support the United Nations' fourth Sustainable Development Goal (UN SDG4) by improving access to quality education through innovative robotics solutions. The robot was tested in a cycle two school (covering Grades 5 to 8 (ages 10 to 13)) in Dubai, United Arab Emirates. It was integrated into math, science, and design technology classes to assess its functionality and efficiency. Surveys conducted among students and teachers showed a high level of acceptance towards the robot, with over 85% of respondents expressing positive attitudes about its presence and interaction in the classroom. However, teachers and students provided feedback concerning the robot's shape, capabilities, and movement mechanism. Teachers also appreciated the robot's alignment with the UN SDG4, stating its capability to support students learning and engagement. The authors highlighted the robot's potential to assist students with sensory challenges, such as hearing and vision impairments, and learning difficulties like dyslexia while emphasizing their commitment to enhancing its accessibility features for a more inclusive learning environment.

Keywords: Humanoid Robot Systems, Intelligent Transportation Systems, AI-Based Methods
Abstract: Recently, LLM-powered driver agents have demonstrated considerable potential in the field of autonomous driving, showcasing human-like reasoning and decision-making abilities. However, current research on aligning driver agent behaviors with human driving styles remains limited, partly due to the scarcity of high-quality natural language data from human driving behaviors. To address this research gap, we propose a multi-alignment framework designed to align driver agents with human driving styles through demonstrations and feedback. Notably, we construct a natural language dataset of human driver behaviors through naturalistic driving experiments and post-driving interviews, offering high-quality human demonstrations for LLM alignment. The framework’s effectiveness is validated through simulation experiments in the CARLA urban traffic simulator and further corroborated by human evaluations. Our research offers valuable insights into designing driving agents with diverse driving styles. The implementation of the framework and details of the dataset can be found at the link.

Keywords: Humanoid Robot Systems, Aerial Systems: Mechanics and Control, Methods and Tools for Robot System Design
Abstract: Co-design optimization strategies usually rely on simplified robot models extracted from CAD. While these models are useful for optimizing geometrical and inertial parameters for robot control, they might overlook important details essential for prototyping the optimized mechanical design. For instance, they may not account for mechanical stresses exerted on the optimized geometries and the complexity of assembly-level design. In this paper, we introduce a co-design framework aimed at improving both the control performance and mechanical design of our robot. Specifically, we identify the robot links that significantly influence control performance. The geometric characteristics of these links are parameterized and optimized using a multi-objective evolutionary algorithm to achieve optimal control performance. Additionally, an automated Finite Element Method (FEM) analysis is integrated into the framework to filter solutions not satisfying the required structural safety margin. We validate the framework by applying it to enhance the mechanical design for flight performance of the jet-powered humanoid robot iRonCub.

Keywords: Modeling, Control, and Learning for Soft Robots, Soft Robot Applications, Dynamics
Abstract: In robotics, while rigid joints are common due to their accuracy and fast response ability, elastic joints are well-known for their safety when interacting with the environment. To harmonise the advantages of these joint types, robots with mixed rigid-elastic joints can be considered. In this paper, a robust-adaptive two-loop control algorithm is proposed to control this type of robot when there are uncertainties in system parameters. In the outer loop, a robust control algorithm is proposed to deal with the uncertainties in the dynamic parameters of the joint side, together with an adaptive controller for the rigid joints. In the inner loop, another robust control algorithm is proposed to handle the uncertainties in system parameters of the elastic joint’s motor side, and a similar adaptive control algorithm is presented to manipulate the elastic joints' motors. The stability of the system is assured by Lyapunov's stability theory. Finally, simulation experiments are conducted to verify the proposed control algorithm.

Keywords: Modeling, Control, and Learning for Soft Robots, Soft Robot Applications
Abstract: Central Pattern Generators (CPG) nonlinear oscillation network is being increasingly used in the control of multijoint collaborative robots. The motion attitude of robots can be effectively adjusted by tuning parameters of the CPG neural network. However, the mapping from CPG parameters to motion attitude is relatively complicated. To improve the motion performance, an optimization method combining computational fluid dynamics (CFD) and CPG network is proposed. In this work, we design a three-joint biomimetic soft robot fish following the body structure of trevally and an improved CPG network based on the Hopf model is incorporated into the control system. Directly optimizing the swimming performance through experiments is time consuming and complex, a mode of first adjusting parameters on the simulation platform and then refining on the robot is usually adopted. Therefore, a CFD simulation platform using hydrodynamic solutions has been established to assist in analyzing the swimming effect after parameters optimization. Finally, the experimental results show that the swimming effect simulated by the CFD simulation platform is highly similar to the real test, and the swimming performance after the improved CPG network optimization has been significantly increased.

Keywords: Modeling, Control, and Learning for Soft Robots, Soft Robot Materials and Design, Soft Robot Applications
Abstract: Soft continuum sections are widely used in robotic mechanisms for achieving dexterous motions. However, most available designs of soft continuum sections cannot support payload during a motion. This paper presents a development of a novel soft robotic wrist section for a prosthetic hand named ‘PRISMA Hand II’. Our research focuses on various phases of development of a soft continuum wrist section, that can support a substantial payload and maintain postures of the hand. Mechanical design, fabrication, and modelling strategies adopted for developing the wrist section are described. The design of the wrist section is constructed by assembling springs, discs, and tendons. The number and dimensions of springs and discs are optimised using static structural analysis. Kinematic modelling and dynamic modelling of the wrist section are carried out using Geometric Variable Strain (GVS) approach based on Cosserat rod theory and a generalised coordinate method respectively. The geometric formulations involved in Cosserat rod theory guaranteed accurate and quick computations considering deformation parameters. Dynamic modelling approach also enhanced performance of the wrist section reducing errors and computational time during real time implementations. This paper also discusses about a dynamic model based controller strategy for the wrist section and advantages of the proposed controller are proved using a comparative study with a kinematic model based PID controller. Experimental validations of motions of the fabricated wrist section employing the dynamic controller are also included in the paper.

Keywords: Modeling, Control, and Learning for Soft Robots, Biomimetics, Soft Robot Applications
Abstract: Recent research on mobile robots has focused on increasing their adaptability to unpredictable and unstructured environments using soft materials and structures. However, the determination of key design parameters and control over these compliant robots are predominantly iterated through experiments, lacking a solid theoretical foundation. To improve their efficiency, this paper aims to provide mathematics modeling over two locomotion, crawling and swimming. Specifically, a dynamic model is first devised to reveal the influence of the contact surfaces’ frictional coefficients on displacements in different motion phases. Besides, a swimming kinematics model is provided using coordinate transformation, based on which, we further develop an algorithm that systematically plans human-like swimming gaits, with maximum thrust obtained. The proposed algorithm is highly generalizable and has the potential to be applied in other soft robots with similar multiple joints. Simulation experiments have been conducted to illustrate the effectiveness of the proposed modeling.

Keywords: Modeling, Control, and Learning for Soft Robots, Soft Robot Materials and Design
Abstract: Soft pneumatic bending actuators (SPBAs) are commonly employed in soft robotics due to their unique characteristics, including safety, low weight, speed, and load capacity. However, the combination of pneumatics with soft materials causes SPBAs to exhibit nonlinearities and infinite degrees of freedom, complicating their dynamic modeling. In this work, we present how the dynamics of SPBAs can be adjusted to a fractional order model (FOM), showing an approach for their empirical identification. We also present a method for designing fractional order controllers (FOCs) for this type of actuators, based on the inversion of the empirical FOM. This modeling and control is applied to a modular SPBA made of a smart hydrogel, which endows the actuators with self-healing, self-adhesion, and self-sensing capabilities.

Keywords: Modeling, Control, and Learning for Soft Robots, Machine Learning for Robot Control
Abstract: Many application domains, e.g., in medicine and manufacturing, can greatly benefit from pneumatic Soft Robots (SRs). However, the accurate control of SRs has remained a significant challenge to date, mainly due to their nonlinear dynamics and viscoelastic material properties. Conventional control design methods often rely on either complex system modeling or time-intensive manual tuning, both of which require significant amounts of human expertise and thus limit their practicality. In recent works, the data-driven method, Automatic Neural ODE Control (ANODEC) has been successfully used to -- fully automatically and utilizing only input-output data -- design controllers for various nonlinear systems in silico, and without requiring prior model knowledge or extensive manual tuning. In this work, we successfully apply ANODEC to automatically learn to perform agile, non-repetitive reference tracking motion tasks in a real-world SR and within a finite time horizon. To the best of the authors' knowledge, ANODEC achieves, for the first time, performant control of a SR with hysteresis effects from only 30 seconds of input-output data and without any prior model knowledge. We show that for multiple, qualitatively different and even out-of-training-distribution reference signals, a single feedback controller designed by ANODEC outperforms a manually tuned PID baseline consistently. Overall, this contribution not only further strengthens the validity of ANODEC, but it marks an important step towards more practical, easy-to-use SRs that can automatically learn to perform agile motions from minimal experimental interaction time.

Keywords: Modeling, Control, and Learning for Soft Robots, Dynamics, Intention Recognition
Abstract: Soft exoskeleton robots have promising potential in walking assistance with comfortable wearing experience. In this study, an exosuit equipped with a twisted string actuator (TSA) is developed to provide powerful driving force and diverse operating modes for hemiplegic patients in daily life. It is challenging to establish the human-robot coupling dynamic model due to the soft structure of the exosuit and tight coupling, precise control and effective assistance are difficult to guaranteed in current exosuits. Considering the impedance characteristics of human-robot interaction, an adaptive impedance control method based on reinforcement learning (RL) is proposed, where human motion intention is utilized to optimize impedance parameters and adjust the robot's operating mode. A nonlinear disturbance observer is proposed to compensate for the effects of model estimation errors, joint friction, and external disturbances. Experimental verification demonstrates the effectiveness and superiority of the robotic system.

Keywords: Modeling, Control, and Learning for Soft Robots, Soft Robot Applications
Abstract: Soft continuum robots, characterized by their inherent compliance and dexterity, are increasingly pivotal in applications requiring delicate interactions with the environment such as the medical field. Despite their advantages, challenges persist in accurately modeling and controlling their shape during interactions with surrounding objects. This is because of the difficulty in modeling the large degrees of freedom in soft-bodied objects that become more active during interactions. In this study, we present a deep visual model to predict the interaction shapes of a soft continuum robot in contact with surrounding objects. By formulating this task as a forward-statics problem, the model uses the initial state images containing the object configuration and future actuation values to predict interactive state images of the robot under this actuation condition. We developed and tested the model in both simulated and physical environments, explored the model's predictive capabilities using monocular and binocular views, and tested the model's generalization ability on different datasets. Our results show that deep learning methods are a promising tool for solving the complex problem of predicting the shape of a soft continuum robot interacting with the environment, requiring no prior knowledge about the system dynamics and explicit mapping of the environment. This study paves the way for future explorations in robot-environment interaction modeling and the development of more adaptable interaction shape control strategies.

Keywords: Modeling, Control, and Learning for Soft Robots, Soft Robot Applications, Machine Learning for Robot Control
Abstract: Soft robots offer more flexibility, compliance, and adaptability than traditional rigid robots. They are also typically lighter and cheaper to manufacture. However, their use in real-world applications is limited due to modeling challenges and difficulties in integrating effective proprioceptive sensors. Large-scale soft robots (approx two meters in length) have greater modeling complexity due to increased inertia and related effects of gravity. Common efforts to ease these modeling difficulties such as assuming simple kinematic and dynamics models also limit the general capabilities of soft robots and are not applicable in tasks requiring fast, dynamic motion like throwing and hammering. To overcome these challenges, we propose a data-efficient Bayesian optimization-based approach for learning control policies for dynamic tasks on a large-scale soft robot. Our approach optimizes the task objective function directly from commanded pressures, without requiring approximate kinematics or dynamics as an intermediate step. We demonstrate the effectiveness of our approach through both simulated and real-world experiments.

Keywords: Soft Robot Applications, Engineering for Robotic Systems, Manufacturing, Maintenance and Supply Chains
Abstract: Since the shape of industrial endoscopes is passively altered according to the contact around it, manual inspection approaches of aeroengines through the inspection ports have unreachable areas, and it's difficult to traverse multistage blades and inspect them simultaneously, which requires engine disassembly or the cooperation of multiple operators, resulting in efficiency decline and increased costs. To this end, this paper proposes a novel continuum manipulator with push-pull multisection structure which provides a potential solution for the disadvantages mentioned above due to its higher flexibility, passability, and controllability in confined spaces. The ultra-slender design combined with a tendon-driven mechanism makes the manipulator acquire enough workspace and more flexible postures while maintaining a light weight. Considering the coupling between the tendons in multisection, a innovative kinematics decoupling control method is implemented, which can realize real-time control in the case of limited computational resources. A prototype is built to validate the capabilities of mechatronic design and the performance of the control algorithm. The experimental results demonstrate the advantages of our continuum manipulator in the in-situ inspection of aeroengines' multistage blades, which has the potential to be a replacement solution for industrial endoscopes.

Keywords: Soft Robot Materials and Design, Soft Robot Applications, Grippers and Other End-Effectors
Abstract: This paper presents a soft actuator inspired by origami and a tunable strain limiting layer, which is proposed for reconfigurable soft robotic grasping. Main structure of the actuator is based on Miura origami which generates extension under pressurized air while a limiting layer with tunable length enables the actuator with different motion patterns. By driving the limiting layer through a servo motor, the range of motion and trajectory of the actuator can be pre-programed and the gripper’s grasping range will be affected accordingly. This paper discusses the design, fabrication, analysis and experimental verification of the actuator. Then grasping performance of the gripper under objects of different shapes, sizes, and weights is experimentally evaluated. The reconfigurable soft gripper can be applied as an end-effector to accomplish adaptive grasping tasks with various targets.

Keywords: Soft Robot Materials and Design, Perception for Grasping and Manipulation
Abstract: Soft robotics has emerged as the standard solution for grasping deformable objects, and has proven invaluable for mobile robotic exploration in extreme environments. However, despite this growth, there are no widely adopted computational design tools that produce quality, manufacturable designs. To advance beyond the diminishing returns of heuristic bio-inspiration, the field needs efficient tools to explore the complex, non-linear design spaces present in soft robotics, and find novel high-performing designs. In this work, we investigate a hierarchical design optimization methodology which combines the strengths of topology optimization and quality diversity optimization to generate diverse and high-performance soft robots by evolving the design domain. The method embeds variably sized void regions within the design domain and evolves their size and position, to facilitating a richer exploration of the design space and find a diverse set of high-performing soft robots. We demonstrate its efficacy on both benchmark topology optimization problems and soft robotic design problems, and show the method enhances grasp performance when applied to soft grippers. Our method provides a new framework to design parts in complex design domains, both soft and rigid.

Keywords: Soft Robot Materials and Design, Soft Robot Applications, Modeling, Control, and Learning for Soft Robots
Abstract: Exploration of various applications is the frontier of research on inflatable robots. We proposed an articulated robots consisting of multiple pneumatic bladder links connected by rolling contact joints called Hillberry joints. The bladder link is made of a double-layered structure of tarpaulin sheet and polyurethane sheet, which is both airtight and flexible in shape. The integration of the Hilberry joint into an inflatable robot is also a new approach. The rolling contact joint allows wide range of motion of ±150 degrees, the largest among the conventional inflatable joints. Using the proposed mechanism for inflatable robots, we demonstrated moving a 500 g payload with a 3-DoF arm and lifting 3.4 kg and 5 kg payloads with 2-DoF and 1-DoF arms, respectively. We also experimented with a single 3-DoF inflatable leg attached to a dolly to show that the proposed structure worked for legged locomotion.

Keywords: Soft Sensors and Actuators
Abstract: Recently, there has been a notable shift towards electronics-free designs, which offer promising integration possibilities with soft robots, reducing reliance on traditional electronics. Despite numerous demonstrations showcasing logical control, contact sensors, and gait control, the conventional quake valve-based design is gradually struggling to meet the demands of electronics-free soft robots with increasingly complex functionalities. Integrating multiple tubes, channels, and valves has led to larger and bulkier overall systems. In this study, we introduce a simple yet powerful electronics-free pneumatic valve that excels in various aspects: it allows for flexible function configurations (operating individually, in pairs, or in larger groups), offers high-frequency synchronous reverse outputs, stores valve status, and ensures efficient maintenance. We believe that this work lays the groundwork for developing straightforward yet highly effective fully autonomous soft robots.

Keywords: Soft Sensors and Actuators, Force and Tactile Sensing
Abstract: Tactile information is the research hotspot of wearable flexible sensors due to its importance and complexity. With the innovation of wearable technology and robotics in healthcare, researchers are increasingly integrating wearable flexible sensors on the front end of robots to reproduce the hand tactile manipulation of human tissues. Therefore, it is hoped to develop a thin-film sensor that can be deployed in a small area to assist robots in surgery and data collection of human tissues. Here we use a one-step injection method to fabricate a novel composite sensor based on liquid metal. By laminating multiple PDMS microfluidic layers, the two parameters of pressure and deformation are measured simultaneously in a decoupled manner. The sensor is small and thin, making it easy to integrate into fingers/robot fingers for assistance. The finger/robot finger exerts pressure on the sensor and the sensor deforms with the material to identify the hardness of the material being touched. Separate performance tests of the two sensors show that the strain and pressure functions are decoupled from each other, and their ratios can identify and classify the hardness of different touched materials (glass, PDMS and silicone). This novel composite sensor we proposed can assist robots in manipulating human tissues during medical surgeries. At the same time, its function in tactile information feedback also has broad applications in medical treatment, rehabilitation and services.

Keywords: Soft Sensors and Actuators, Soft Robot Materials and Design, Human-Robot Collaboration
Abstract: Tactile sensors are crucial in robotics as they enable robots to perceive and interact with their environment through touch, akin to the human sense of touch. Adjustable sensors that can adapt to various tasks by functional or structural modification have not been extensively explored. In terms of sensing adjustability of a sensor, two important aspects are the sensor’s sensitivity and compliance. This paper proposes a novel design for an adjustable compliance and sensitivity sensor composed of a silicone base, a permanent magnetic elastomer (PME), and a printed circuit board (PCB) with magnetic transducers installed. Its adjustability is achieved by varying the pneumatic pressure. This paper presents the design, manufacturing process, and experimental characterization of such an adjustable compliance and sensitivity sensor. This paper thoroughly investigates how altering the pressure of the sensor influences its sensing properties. The results show that it can achieve adjustability in all three axes. For the current design, the sensitivity can be varied from 0.093 to 0.125 mT/N (34.41%), 0.089 to 0.13 mT/N (31.54%), and 0.169 to 0.45 mT/N (62.44%) in the X-, Y-, and Z-axis, respectively. The deformation it undertakes varies from 3.20 to 3.79 mm (18.44%), indicating the compliance change.

Keywords: Intention Recognition, Deep Learning Methods, Sensor-based Control
Abstract: Upper limb amputees face significant challenges in their daily lives due to the loss of hand or arm functionality. Researchers have developed upper limb prostheses to restore normal hand movements for them. Most hand movement recognition systems of prostheses use electromyography (EMG) as the input signal source, but ignore the interrelationship with electroencephalography (EEG), which may contain valuable movement-related information as well. In order to enhance the accuracy of hand movement classification, we proposed a hand movement recognition system based on a one-dimensional convolutional neural network (1D-CNN) that combines EEG and EMG as the input signal sources to increase the quantity of accessible information. In this work, we collected the EEG and EMG of five subjects during the hand movements and used a 1D-CNN based model to classify the preprocessed signals. The average accuracy of using EEG-EMG fusion is 96.59±2.63%, significantly higher than 74.99±8.24% of using single EEG and 90.31±7.16% of using single EMG. Then, we applied the model trained by offline experiment for online recognition, and controlled the Pepper robot to complete the corresponding hand movements. The average accuracy of online recognition can reach 93.00±4.85% by using majority voting method. The results indicate that the method of EEG-EMG fusion can effectively enhance the performance of hand movement recognition system, which promote the development of upper limb prostheses and contribute to the rehabilitation of upper limb amputees.

Keywords: Brain-Machine Interfaces, Telerobotics and Teleoperation, Human Factors and Human-in-the-Loop
Abstract: To address these challenges, our method estimates the user’s intended goal from their commands and uses this goal to generate auxiliary commands through the autonomous system that are both at a higher input frequency and more continuous.Furthermore, by defining the confidence level of the estimation, we adaptively calculated the weights for combining user and autonomous commands, thus achieving shared control. We conducted navigation experiments in both simulated environments and participant experiments in real environments including user ratings, using a pseudo-BMI setup. As a result, the proposed method significantly reduced obstacle collisions in all experiments. It markedly shortened path lengths under almost all conditions in simulations and, in participant experiments, especially when user inputs become more discrete and noisy (p<0.01). Furthermore, under such challenging conditions, it was demonstrated that users could operate more easily, with greater confidence, and at a comfortable pace through this system.

Keywords: Cognitive Control Architectures, Model Learning for Control, Intelligent Transportation Systems
Abstract: This study introduces a novel approach utilizing Gaussian process model predictive control (MPC) to stabilize the output voltage of a polymer electrolyte fuel cell (PEFC) system by simultaneously regulating hydrogen and airflow rates. Two Gaussian process models are developed to capture PEFC dynamics, taking into account constraints including hydrogen pressure and input change rates, thereby aiding in mitigating errors inherent to PEFC predictive control. The dynamic performance of the physical model and Gaussian process MPC in constraint handling and system inputs is compared and analyzed. Simulation outcomes demonstrate that the proposed Gaussian process MPC effectively maintains the voltage at the target 48 V while adhering to safety constraints, even amidst workload disturbances ranging from 110-120 A.

Keywords: Cognitive Control Architectures, Biologically-Inspired Robots, Autonomous Agents
Abstract: Autonomous Mobile Robots are used with increasing frequency in inspection and maintenance tasks completing fixed goal sequences. The downtime robots experience between goals offers an opportunity to gather additional environment information instead of resting. Uncertainty in the amount of downtime available rules out the definition of a pre-determined schedule set by an external operator. Instead, the robot itself should decide dynamically, what information it should gather before its next task begins. This results in a multi-objective optimization problem trying to maximize information gain while utilizing as much of the available time as possible. We propose a genetic algorithm to solve the presented optimization problem and introduce two different models for artificial curiosity used inside the fitness function for gathering as much information as possible. For planning the genetic algorithm utilizes a multi-map approach using information and obstacle maps. We evaluated our models in a pre-defined and pre-mapped Gazebo environment with a given information map and evaluated their performance against an information-agnostic coverage algorithm. In this work, we show that utilizing artificial curiosity in path planning can result in major information gains by effectively using downtime.

Keywords: Cognitive Modeling, Neurorobotics, Bioinspired Robot Learning
Abstract: We present ROBOVERINE, a neural dynamic robotic active vision process model of selective visual attention and scene grammar in naturalistic environments. The model addresses significant challenges for cognitive robotic models of visual attention: combined bottom-up salience and top-down feature guidance, combined overt and covert attention, coordinate transformations, two forms of inhibition of return, finding objects outside of the camera frame, integrated space- and object-based analysis, minimally supervised few-shot continuous online learning for recognition and guidance templates, and autonomous switching between exploration and visual search. Furthermore, it incorporates a neural process account of scene grammar - prior knowledge about the relation between objects in the scene - to reduce the search space and increase search efficiency. The model also showcases the strength of bridging two frameworks: Deep Neural Networks for feature extractions and Dynamic Field Theory for cognitive operations.

Keywords: Human Factors and Human-in-the-Loop, Reinforcement Learning, Cognitive Modeling
Abstract: Large Language Models (LLMs) are increasingly influential in advancing robotics. This paper introduces ECLAIR (Evaluative Corrective Guidance Language as Reinforcement), a novel framework that leverages LLMs to interpret and incorporate diverse natural language feedback into robotic learning. ECLAIR unifies various forms of human advice into actionable insights within a Reinforcement Learning context, enabling more efficient robot instruction. Experiments with real-world users demonstrate that ECLAIR accelerates the robot's learning process, aligning its policy closer to optimal from the outset and reducing the need for extensive human intervention. Additionally, ECLAIR effectively integrates multiple types of advice and adapts well to prompt modifications. It also supports multilingual instruction, broadening its applicability and fostering more inclusive human-robot interactions. Project website: https://sites.google.com/view/eclairiros

Keywords: Human and Humanoid Motion Analysis and Synthesis, Modeling and Simulating Humans, Human-Centered Robotics
Abstract: This paper presents a novel method for detecting and estimating contact forces only from human motions using machine learning techniques. Knowing the location of the contacts with the environment and the magnitude of the exerted force is critical for dynamic human motion analysis. However, their annotation is usually made manually from captured motion data especially in case of multiple contacts even if the data includes force measurement. Moreover, most existing human motion datasets do not include contact force. To overcome these bottlenecks, we introduce a network that leverages vector-quantized variational autoencoder (VQ-VAE) and self-attention that learns a small set of discrete feature values representing various contact states. These feature values, called contact codes, allow human motions to be converted to contact states and resulting forces. By applying an optimization for contact estimation with a reduced set of manual annotations, the existence of contacts can be automatically determined, which is essential information for dynamic analysis. We validated the effectiveness and potential usefulness of the proposed method with a human walking gait dataset, by converting the human motions into contact sequences and forces and applying the estimated contacts to dynamic motion analysis.

Keywords: Human and Humanoid Motion Analysis and Synthesis, Modeling and Simulating Humans, Human Detection and Tracking
Abstract: Human mesh reconstruction (HMR) from monocular video plays an important role in human-robot interaction and collaboration. However, existing video-based human mesh reconstruction methods face a tradeoff between accurate reconstruction and smooth motion. These methods design networks based on either RNNs or attention mechanisms to extract local temporal correlations or global temporal dependencies, but the lack of complementary longterm information and local details limits their representation of the human body. To address this problem, we propose a Dual-branch Graph Transformer network for 3D human mesh Reconstruction from video, named DGTR. DGTR employs a dual-branch network including a Global Motion Attention (GMA) branch and a Local Details Refine (LDR) branch to parallelly extract long-term dependencies and local crucial information, helping model global human motion and local human details (e.g., local motion, tiny movement). Specifically, GMA utilizes a global transformer to model long-term human motion. LDR combines modulated graph convolutional networks and the transformer framework to aggregate local information in adjacent frames and extract crucial information of human details. Experiments demonstrate that our DGTR outperforms state-of-the-art video-based methods in reconstruction accuracy and maintains competitive motion smoothness. Moreover, DGTR utilizes fewer parameters and FLOPs, which validate the effectiveness and efficiency of the proposed DGTR. Code is publicly available at https://github.com/TangTao-PKU/DGTR.

Keywords: Human and Humanoid Motion Analysis and Synthesis, Intention Recognition, Modeling and Simulating Humans
Abstract: Long-term human trajectory prediction is a challenging yet critical task in robotics and autonomous systems. Prior work that studied how to predict accurate short-term human trajectories with only unimodal features often failed in long-term prediction. Reinforcement learning provides a good solution for learning human long-term behaviors but can suffer from challenges in data efficiency and optimization. In this work, we propose a long-term human trajectory forecasting framework that leverages a guided diffusion model to generate diverse long-term human behaviors in a high-level latent action space, obtained via a hierarchical action quantization scheme using a VQ-VAE to discretize continuous trajectories and the available context. The latent actions are predicted by our guided diffusion model, which uses physics-inspired guidance at test time to constrain generated multimodal action distributions. Specifically, we use reachability analysis during the reverse denoising process to guide the diffusion steps toward physically feasible latent actions. We evaluate our framework on two publicly available human trajectory forecasting datasets: SFU-Store-Nav and JRDB, and extensive experimental results show that our framework achieves superior performance in long-term human trajectory forecasting.

Keywords: Human Detection and Tracking, Human-Robot Collaboration, Computer Vision for Manufacturing
Abstract: In the context of robotics, accurate 3D human pose estimation is essential for enhancing human-robot collaboration and interaction. This manuscript introduces a multi-view 2D to 3D lifting optimization-based method designed for video-based 3D human pose estimation, incorporating temporal information. Our technique addresses key challenges, namely robustness to 2D joint detection error, occlusions, and varying camera perspectives. We evaluate the performance of the algorithm through extensive experiments on the MPI-INF-3DHP dataset. Our method demonstrates very good robustness up to 25 pixels of 2D joint error and shows resilience in scenarios involving several occluded joints. Comparative analyses against existing 2D to 3D lifting and multi-view methods showcase good performance of our approach.

Keywords: Intention Recognition
Abstract: With the increased importance of autonomous navigation systems has come an increasing need to protect the safety of Vulnerable Road Users (VRUs) such as pedestrians. Predicting pedestrian intent is one such challenging task, where prior work predicts the binary cross/no-cross intention with a fusion of visual and motion features. However, there has been no effort so far to hedge such predictions with human-understandable reasons. We address this issue by introducing a novel problem setting of exploring the intuitive reasoning behind a pedestrian’s intent. In particular, we show that predicting the ‘WHY’ can be very useful in understanding the ‘WHAT’. To this end, we propose a novel, reason-enriched PIE++ dataset consisting of multi-label textual explanations/reasons for pedestrian intent. (Explanations, in our context, refers to the interpretation of pedestrian intent and not model interpretability.) We also introduce a novel multi-task learning framework called MINDREAD, which leverages a cross-modal representation learning framework for predicting pedestrian intent as well as the reason behind the intent. Our comprehensive experiments show significant improvement of 5.6% and 7% in accuracy and F1-score for the task of intent prediction on the PIE++ dataset using MINDREAD. We also achieved a 4.4% improvement in accuracy on a commonly used JAAD dataset. Extensive evaluation using quantitative/qualitative metrics and user studies show the effectiveness of our approach

Keywords: Human-Centered Automation, Intention Recognition, Humanoid Robot Systems
Abstract: Robotic assistance is a current research topic with high application value and multiple challenges. Assistive robots are used in various scenarios, such as production lines, operating tables, and elderly care. While providing effective assistance, most of the assistance tasks that current robots can perform are limited to predefined tasks. This limitation arises from the insufficiency of the current robot perception system to forecast future human activities. To address this issue, we propose a novel 2-stage robotic assistant for human activities through future human-object interaction (HOI) segment prediction. Unlike previous work focusing on predefined or short-term tasks, our robotic assistant can make predictions for future assistance according to human habits. In the first stage, we propose a visual-based human-object interaction segment prediction method to predict human activities, which enables the robotic system to infer human intention. Moreover, we define the robotic executable tasks as an interactive tuple to keep the robotic assistance normatively consistent with human activity. Meanwhile, a graph convolutional network with geometric features that can predict human-object interaction segments is proposed to provide target manipulation and target object for the assistive robot. In the second stage, we present a mobile task completion process including visual navigation, object localization and grasping. The perception stage is evaluated on the MPHOI dataset and custom-collected SPHOI dataset. Finally, we evaluate our comprehensive framework through real-time experimentation.

Keywords: Human-Robot Collaboration, Telerobotics and Teleoperation, Intention Recognition
Abstract: Assistive robot arms can help humans by partially automating their desired tasks. Consider an adult with motor impairments controlling an assistive robot arm to eat dinner. The robot can reduce the number of human inputs — and how precise those inputs need to be — by recognizing what the human wants (e.g., a fork) and assisting for that task (e.g., moving towards the fork). Prior research has largely focused on learning the human’s task and providing meaningful assistance. But as the robot learns and assists, we also need to ensure that the human understands the robot’s intent (e.g., does the human know the robot is reaching for a fork?). In this paper, we study the effects of communicating learned assistance from the robot back to the human operator. We do not focus on the specific interfaces used for communication. Instead, we develop experimental and theoretical models of a) how communication changes the way humans interact with assistive robot arms, and b) how robots can harness these changes to better align with the human’s intent. We first conduct online and in-person user studies where participants operate robots that provide partial assistance, and we measure how the human’s inputs change with and without communication. With communication, we find that humans are more likely to intervene when the robot incorrectly predicts their intent, and more likely to release control when the robot correctly understands their task. We then use these findings to modify an established robot learning algorithm so that the robot can correctly interpret the human’s inputs when communication is present. Our results from a second in-person user study suggest that this combination of communication and learning outperforms assistive systems that isolate either learning or communication.

Keywords: Multi-Modal Perception for HRI, RGB-D Perception, Data Sets for Robotic Vision
Abstract: Human activity recognition is a vital area of robotics with significant real-world applications, from enhancing security and surveillance to improving healthcare and human-robot interaction. A critical challenge lies in bridging the gap between research models, which often assume ideal conditions, and the complexities of real-world environments. In practice, conditions can be far from perfect, including scenarios with poor lighting, adverse weather, or blurred views. In this paper, we present an innovative approach for robust activity recognition through learned sensor fusion, in which our recognition framework identifies a latent weighted combination of input modalities, enabling classifiers to capitalize on advantages provided by various sensors. In support of our work, we have released a dataset of human activities across multiple modalities with environmental degradation factors such as darkness, fog, and thermal blur. Our proposed approach identifies a weighted combination of modality representations derived from existing architectures. We show that our approach is able to achieve 24% higher classification performance than existing single-modality approaches. Our approach also attains comparable performance to modality fusion approaches in significantly reduced classification time. In real-world robotics applications, particularly those occurring in dangerous, degraded environments, this speed is critical.

Keywords: Robot Companions, Human-Centered Automation, Human Detection and Tracking
Abstract: Reliable Human Orientation Estimation (HOE) from a monocular image is critical for autonomous agents to understand human intention. Significant progress has been made in HOE under full observation. However, the existing methods easily make a wrong prediction under partial observation and give it an unexpectedly high confidence. To solve the above problems, this study first develops a method called Part-HOE that estimates orientation from the visible joints of a target person so that it is able to handle partial observation. Subsequently, we introduce a confidence-aware orientation estimation method, enabling more accurate orientation estimation and reasonable confidence estimation under partial observation. The effectiveness of our method is validated on both public and custom-built datasets, and it shows great accuracy and reliability improvement in partial observation scenarios. In particular, we show in real experiments that our method can benefit the robustness and consistency of the Robot Person Following (RPF) task.

Keywords: Surveillance Robotic Systems, Visual Tracking
Abstract: In this paper, we develop a multiple-object zoom- ing system which can capture clear images of different objects at an ultrafast speed. The system consists of a panoramic HFR stereo camera and a galvanometer-based reflective pan- tilt-zoom (PTZ) camera. In order to alleviate the impact of brightness, noise, and viewing angle in the image, we use the motion information of the object for stereo correspondence. According to the spatial positions of all objects obtained from HFR stereo correspondence, we can obtain the control voltage of the pan and tilt mirror of the galvanometer-based reflective PTZ camera through the mapping relationship. Then, PTZ camera captures clear images of multiple objects in a time-division multiplexed manner at an extremely fast speed. Experimental results show that we can distinguish multiple fast- moving people indoors in the HFR stereo camera and capture their high-definition facial images simultaneously.

Keywords: Computer Vision for Automation, Vision-Based Navigation, Semantic Scene Understanding
Abstract: Developing a reliable vision system is a fundamental challenge for robotic technologies (e.g., indoor service robots and outdoor autonomous robots) which can ensure reliable navigation even in challenging environments such as adverse weather conditions (e.g., fog, rain), poor lighting conditions (e.g., over/under exposure), or sensor degradation (e.g., blurring, noise), and can guarantee high performance in safety-critical functions. Current solutions proposed to improve model robustness usually rely on generic data augmentation techniques or employ costly test-time adaptation methods. In addition, most approaches focus on addressing a single vision task (typically, image recognition) utilising synthetic data. In this paper, we introduce Per-corruption Adaptation of Normalization statistics (PAN) to enhance the model robustness of vision systems. Our approach entails three key components: (i) a corruption type identification module, (ii) dynamic adjustment of normalization layer statistics based on identified corruption type, and (iii) real-time update of these statistics according to input data. PAN can integrate seamlessly with any convolutional model for enhanced accuracy in several robot vision tasks. In our experiments, PAN obtains robust performance improvement on challenging real-world corrupted image datasets (e.g., OpenLoris, ExDark, ACDC), where most of the current solutions tend to fail. Moreover, PAN outperforms the baseline models by 20-30% on synthetic benchmarks in object recognition tasks.

Keywords: Computer Vision for Automation, Perception for Grasping and Manipulation, Computer Vision for Manufacturing
Abstract: Bin picking systems in factory automation usually face robustness issues caused by sparse and noisy 3D data of metallic objects. Utilizing multiple views, especially with a one-shot 3D sensor and “sensor on hand” configuration is getting more popularity due to its effectiveness, flexibility, and low cost. While moving the 3D sensor to acquire multiple views for 3D fusion, joint optimization, or active vision suffers from low-speed issues. That is because sensing is taken as a decoupled module from motion tasks and is not intentionally designed for a bin picking system. To address the problems, we designed a bin picking system, which tightly couples a multi-view, active vision scheme with motion tasks in a “sensor on hand” configuration. It not only speeds up the system by parallelizing the high-speed sensing scheme to the robot place action but also decides the next sensing path to maintain the continuity of the whole picking process. Unlike others focusing only on sensing evaluation, we also evaluated our design by picking experiments on 5 different types of objects without human intervention. Our experiments show the whole sensing scheme can be finished within 1.682 seconds (maximum) on CPU and the average picking complete rate is over 97.75%. Due to the parallelization with robot motion, the sensing scheme accounts for only 0.635 seconds in takt time on average.

Keywords: Computer Vision for Automation, Object Detection, Segmentation and Categorization, Deep Learning for Visual Perception
Abstract: Monocular 3D object detection plays a crucial role in autonomous driving. However, existing monocular 3D detection algorithms depend on 3D labels derived from LiDAR measurements, which are costly to acquire for new datasets and challenging to deploy in novel environments. Specifically, this study investigates the pipeline for training a monocular 3D object detection model on a diverse collection of 3D and 2D datasets. The proposed framework comprises three components: (1) a robust monocular 3D model capable of functioning across various camera settings, (2) a selective-training strategy to accommodate datasets with differing class annotations, and (3) a pseudo 3D training approach using 2D labels to enhance detection performance in scenes containing only 2D labels. With this framework, we could train models on a joint set of various open 3D/2D datasets to obtain models with significantly stronger generalization capability and enhanced performance on new dataset with only 2D labels. We conduct extensive experiments on KITTI, nuScenes, ONCE, Cityscapes and BDD100K datasets to demonstrate the scaling ability of the proposed method.

Keywords: Computer Vision for Automation, Visual Tracking, RGB-D Perception
Abstract: In this paper, a method for 3D non-rigid motion estimation of a surface using an RGB-D camera in eye-in-hand configuration is presented. The eye-in-hand configuration eliminates errors typically associated with camera-end-effector calibration, and is thus desirable for task on moving surfaces such as bioprinting. However, its implementation is challenging since camera and surface of interest are moving, making mesh-based approaches unsuitable. Thus, the proposed method operates directly on point clouds, benefiting from accurate and simplified data processing. A point cloud contains both intensity and depth data, with the former used to estimate in-plane deformation and the latter to compute full 3D deformation. Surface deformation is modeled via a Thin Plate Spline model. The method accuracy is assessed at 0.1 mm accuracy in simulated datasets, rendering it suitable for precision tasks, and its feasibility is validated experimentally on a moving platform that deforms at a rate of 0.8 Hz with a 4 mm in-plane amplitude and a 20 mm elevation amplitude.

Keywords: Computer Vision for Automation, Computer Vision for Transportation, Object Detection, Segmentation and Categorization
Abstract: Despite their success in LIDAR object detection, modern detectors are vulnerable to uncommon instances and corner cases (e.g., a runaway tire) since they are closed-set and static. Networks under the closed-set setup only predict labels of seen classes, while static models suffer from catastrophic forgetting when gradually learning novel concepts. This motivates us to formulate the open-world 3D object detection task for autonomous driving, which aims to 1) tackle the closed-set issue by identifying unseen instances as unknown and 2) incrementally learn novel classes without forgetting previously obtained knowledge. To achieve the open-world objectives, we propose Open-World 3D Detector (OW3Det), the first framework for open-world 3D object detection. The OW3Det comprises a base detector, a self-supervised unknown identifier, and a knowledge-distillation-restricted incremental learner. Although knowledge distillation facilitates preserving memories, imposing penalties on areas containing unknown objects hinders the incremental learning process. We mitigate this hindrance by employing unknown-driven pivotal mask, which eliminates unnecessary restrictions on regions overlapping with novel instances. Abundant experiments and visualizations demonstrate that the proposed OW3Det attains state-of-the-art performance.

Keywords: Computer Vision for Automation, Hardware-Software Integration in Robotics
Abstract: UAVs and other fast moving robots often need to keep track of distant objects. Conventional zoom cameras commit to a particular viewpoint, and carrying multiple zoom cameras for multi-object tracking is not feasible for power limited robotic systems. We present a dual camera setup that allows tracking of multiple targets at nearly 1km distance with high-resolution. Our setup includes a wide angle camera providing a conventional resolution view and a MEMS driven zoom camera that can query a specific region within the wide angle camera (WAC). We built and calibrated the two-camera system and implemented a real-time image fusion pipeline. We show multi-object tracking and stabilization in real world scenarios.

Keywords: Computer Vision for Automation, Mapping, SLAM
Abstract: Walk2Map has emerged as a promising data-driven method to generate indoor traversability maps based solely on pedestrian trajectories, offering great potential for indoor robot navigation. In this study, we investigate a novel approach, referred to as Walk2Map++, which involves replacing Walk2Map’s first-person sensor (i.e., IMU) with a human observing third-person view from the robot’s onboard camera. However, human observation from a third-person camera is significantly ill-posed due to visual uncertainties resulting from occlusion, nonlinear perspective, depth ambiguity, and human-to-human interaction. To regularize the ill-posedness, we propose to integrate two types of constraints: photometric (i.e., occlusion ordering) and physical (i.e., collision avoidance). We demonstrate that these constraints can be effectively inferred from the interaction between past and present observations, human trackers, and object reconstructions. We depict the seamless integration of asynchronous map optimization events, like loop closure, into the real-time traversability map, facilitating incremental and efficient map refinement. We validate the efficacy of our enhanced methodology through rigorous fusion and comparison with established techniques, demonstrating its capability to advance traversability prediction in complex indoor environments. The code and datasets associated with this study will be made publicly available upon acceptance, facilitating further research and adoption in the field.

Keywords: Computer Vision for Automation, Perception for Grasping and Manipulation
Abstract: Precise manipulation that is generalizable across scenes and objects remains a persistent challenge in robotics. Current approaches for this task heavily depend on having a significant number of training instances to handle objects with pronounced visual and/or geometric part ambiguities. Our work explores the grounding of fine-grained part descriptors for precise manipulation in a zero-shot setting by utilizing web-trained text-to-image diffusion-based generative models. We tackle the problem by framing it as a dense semantic part correspondence task. Our model returns a gripper pose for manipulating a specific part, using as reference a user-defined click from a source image of a visually different instance of the same object. We require no manual grasping demonstrations as we leverage the intrinsic object geometry and features. Practical experiments in a real-world tabletop scenario validate the efficacy of our approach, demonstrating its potential for advancing semantic-aware robotics manipulation.

Keywords: Computer Vision for Automation, Computer Vision for Medical Robotics, Object Detection, Segmentation and Categorization
Abstract: Bronchoscopy reveals air passages and internal tissues for accurate diagnosis of various lung diseases. Robot-assisted bronchoscopy using an airway tree model can help path planning before surgery and navigation during surgery. In airway tree modeling, though volumetric deep learning methods have achieved good performance for airway segmentation, it remains a challenge due to the breakages and leakages. Some existing methods adopt post-processing using traditional methods like morphological and fuzzy connected algorithms. Also, some methods convert the volumetric data to point cloud format to refine segmentation. In this paper, we develop a new point cloud-based approach to refine volumetric segmentation. To address the breakage issue, we approach it as a regression problem of the branch extension direction and length. To tackle the leakage issue, we approach it as a segmentation task to eliminate leakages caused by breakage filling and from volumetric segmentation. Moreover, the direction information of branches is crucial for constructing the airway tree while point clouds do not naturally encode it. To introduce this information, we propose a directional feature aggregation, which first decomposes features of neighboring points based on their locations and aggregates decomposed features to aid the network in capturing the directional information effectively. Our proposed model has been evaluated on two public datasets, and the results show that our refinement can improve the volumetric segmentation.

Keywords: Computer Vision for Automation, Simulation and Animation, Calibration and Identification
Abstract: Fluid interactions are crucial in daily tasks, with properties like density and viscosity being key parameters. The property states can be used as control signals for robot operation. While density estimation is simple, assessing viscosity, especially for different fluid types, is complex. This study introduces a novel differentiable fitting framework, DiffStir, tailored to identify key physics parameters through stirring. Then, given the estimated physics parameters, we can generate commands to guide the robotic stirring. Comprehensive experiments were conducted to validate the efficacy of DiffStir, showcasing its precision in parameter estimation when benchmarked against reported values in the literature. More experiments and videos can be found in the supplementary materials and on the website: url{https://diffstir.robotflow.ai}.

Keywords: Computer Vision for Automation, Deep Learning for Visual Perception, Autonomous Vehicle Navigation
Abstract: 3D single object tracking (SOT) based on point cloud has attracted much attention due to its important role in machine vision and autonomous driving. Recently, textbf{M^2} -Track propose a two-stage tracking structure centered on motion, but they ignore the effect of segmentation errors in sparse point cloud scenarios, which hinder the ability of networks to accurately represent tracking targets. To solve the problems, we propose an efficient 3D single object tracker (Abbr. EST) that can effectively segment point cloud features. Firstly, the proposed fusion segmentation module makes up for the feature loss caused by the downsampling strategy and enhances the ability of the network to recognize foreground points. In addition, the global embedded module is used to further focus on the crucial features of the target. This module provides global information by using residual networks and adding background information. Numerous experiments conducted on KITTI and NuScenes benchmarks show that EST achieves superior point cloud tracking in both performance and efficiency.

Keywords: Computer Vision for Automation, Deep Learning for Visual Perception, RGB-D Perception
Abstract: This research paper presents an innovative multi-task learning framework that allows concurrent depth estimation and semantic segmentation using a single camera. The proposed approach is based on a shared encoder-decoder architecture, which integrates various techniques to improve the accuracy of the depth estimation and semantic segmentation task without compromising computational efficiency. Additionally, the paper incorporates an adversarial training component, employing a Wasserstein GAN framework with a critic network, to refine model's predictions. The framework is thoroughly evaluated on two datasets - the outdoor Cityscapes dataset and the indoor NYU Depth V2 dataset - and it outperforms existing state-of-the-art methods in both segmentation and depth estimation tasks. We also conducted ablation studies to analyze the contributions of different components, including pre-training strategies, the inclusion of critics, the use of logarithmic depth scaling, and advanced image augmentations, to provide a better understanding of the proposed framework.

Keywords: Computer Vision for Automation, Sensor Fusion, Robotics and Automation in Life Sciences
Abstract: This paper introduces a framework for monocular 3D reconstruction of cheetah movements, leveraging a combination of data-driven and physics-based modeling as well as trajectory optimization. Unlike traditional methods that rely solely on kinematics, our approach integrates dynamic motion principles, enhancing the plausibility and generalization of motion estimates. Validated on the cheetah running dataset, AcinoSet, we achieve mean per-joint position errors of 78.8 mm and 72.5 mm, showcasing significant advancements over the existing model used in AcinoSet. By addressing the challenge of absent ground truth data, this work not only advances animal motion capture techniques but also informs the development of bio-inspired robotic systems, offering a robust solution for accurately capturing complex animal locomotion in natural settings.

Keywords: Computer Vision for Automation, Deep Learning for Visual Perception, Industrial Robots
Abstract: The estimation of the 6D pose of industrial components is essential for smart manufacturing. Especially for complex units that require intensive manual operations, such as a concentrator photovoltaics solar panel, accurate spatial localization provides visual aids for industrial automation. In this paper, we propose an accurate and scalable framework to address the dimensional variability of industrial components and tackle practical implementation issues. First, we use the scalable architecture EfficientNet as the backbone coupled with an enhanced feature pyramid network to estimate the object’s pose. By introducing vertical and horizontal connections of shallow layers, the feature extraction of small objects is optimized for better detection accuracy. Second, leveraging the reliable 2D detection results and geometry information, an adaptive pose refinement module is designed to adjust the estimated 6D pose. The scaling of the backbone network and the computational complexity of refined modules are uniformly adjusted via a shared hyperparameter, resulting in a globally scalable framework. In terms of the pose estimation accuracy, the effectiveness of the refinement module and the real-time performance, validations are conducted both on the LINEMOD dataset and our customized datasets comprising of objects from the industrial photovoltaic system. Additionally, to further illustrate the effectiveness of the proposed method, a precision parallel robot is employed to validate the accuracy of real-time object pose tracking.

Keywords: Computer Vision for Automation, Deep Learning for Visual Perception, Data Sets for Robotic Vision
Abstract: Few-shot object detection has drawn increasing attention in the field of robotic exploration, where robots are required to find unseen objects with a few online provided examples. Despite recent efforts have been made to yield online processing capabilities, slow inference speeds of low-powered robots fail to meet the demands of real-time detection-making them impractical for autonomous exploration. Existing methods still face performance and efficiency challenges, mainly due to unreliable features and exhaustive class loops. In this work, we propose a new paradigm AirShot, and discover that, by fully exploiting the valuable correlation map, AirShot can result in a more robust and faster few-shot object detection system, which is more applicable to robotics community. The core module Top Prediction Filter (TPF) can operate on multi-scale correlation maps in both the training and inference stages. During train- ing, TPF supervises the generation of a more representative correlation map, while during inference, it reduces looping iterations by selecting top-ranked classes, thus cutting down on computational costs with better performance. Surprisingly, this dual functionality exhibits general effectiveness and efficiency on various off-the-shelf models. Exhaustive experiments on COCO2017, VOC2014, and SubT datasets demonstrate that TPF can significantly boost the efficacy and efficiency of most off-the-shelf models, achieving up to 36.4% precision improve- ments along with 56.3% faster inference speed. Code and Data are at: https://github.com/ImNotPrepared/AirShot.

Keywords: Visual Tracking, Computer Vision for Automation, Vision-Based Navigation
Abstract: Vision-based self-localization is a crucial technology for enabling autonomous robot navigation in GPS-deprived environments. However, standard frame cameras are subject to motion blur and suffer from a limited dynamic range. This research focuses on efficient feature tracking for self-localization by using event-based cameras. Such cameras do not provide regular snapshots of the environment but asynchronously collect events that correspond to a small delta of illumination in each pixel independently, thus addressing the issue of motion blur during fast motion and high dynamic range. Specifically, we propose a continuous real-time asynchronous event-based feature tracking pipeline, named RATE. This pipeline integrates (i) a corner detector node utilizing a time slice of the Surface of Active Events to initialize trackers continuously, along with (ii) a tracker node with a proposed tracking manager, consisting of a grid-based distributor to reduce redundant trackers and to remove feature tracks of poor quality. Evaluations using public datasets reveal that our method maintains a stable number of tracked features, and performs real-time tracking efficiently while maintaining or even improving tracking accuracy compared to state-of-the-art event-only tracking methods. Our ROS implementation is released as open-source: https://github.com/mikihiroikura/RATE

Keywords: Dynamics, AI-Based Methods, Field Robots
Abstract: Motion prediction is critical for autonomous off-road driving, however, it presents significantly more challenges than on-road driving because of the complex interaction between the vehicle and the terrain. Traditional physics-based approaches encounter difficulties in accurately modeling dynamic systems and external disturbance. In contrast, data-driven neural networks require extensive datasets and struggle with explicitly capturing the fundamental physical laws, which can easily lead to poor generalization. By merging the advantages of both methods, neuro-symbolic approaches present a promising direction. These methods embed physical laws into neural models, potentially significantly improving generalization capabilities. However, no prior works were evaluated in real-world settings for off-road driving. To bridge this gap, we present PhysORD, a neural-symbolic approach integrating the conservation law, i.e., the Euler-Lagrange equation, into data-driven neural models for motion prediction in off-road driving. Our experiments showed that PhysORD can accurately predict vehicle motion and tolerate external disturbance by modeling uncertainties. It outperforms existing methods both in accuracy and efficiency and demonstrates data-efficient learning and generalization ability in long-term prediction.

Keywords: Field Robots, Motion and Path Planning, Motion Control
Abstract: Bridge inspections are essential for maintaining key infrastructure and preventing structural and functional failures. Nevertheless, traditional manual inspection techniques are plagued by laboriousness, high risk, and low efficiency. Although numerous automation inspection methods have been studied, inspection performance remains challenging. The main difficulties are redundant mechanisms, complex control, high energy consumption, and limited autonomy and safety. To address these problems, we are developing a small, lightweight, electrically-driven robotic manipulator for bridge inspection named the BIRM. Here, we propose a kinetic-energy-optimal and safety-guaranteed trajectory planning for BIRM. Compared with existing methods, it simultaneously addresses energy consumption and safety. The approach formulates a quadratic programming (QP) problem by considering the robot's kinetic energy as the objective function, and the augmented Lagrange multiplier (ALM) is applied to find the solution of the QP. The proposed method completely satisfies the joint position, velocity, and acceleration limits at the speed level while considering collision avoidance. In this paper, the collision detection strategy can achieve low-complexity computation through several structural parameters of the bridge, thereby quickly adapting to environmental changes. Through simulation experiments, we validate the effectiveness and superiority of the proposed method. Through physical experiments, we demonstrate the sustainability and safety of bridge inspections in the field.

Keywords: Field Robots, Energy and Environment-Aware Automation, Deep Learning Methods
Abstract: Recent works in field robotics highlighted the importance of resiliency against different types of terrains. Boreal forests, in particular, are home to many mobility-impeding terrains that should be considered for off-road autonomous navigation. Also, being one of the largest land biomes on Earth, boreal forests are an area where autonomous vehicles are expected to become increasingly common. In this paper, we address the issue of classifying boreal terrains by introducing BorealTC, a publicly available dataset for proprioceptive-based terrain classification (TC). Recorded with a Husky A200, our dataset contains 116 min of Inertial Measurement Unit (IMU), motor current, and wheel odometry data, focusing on typical boreal forest terrains, notably snow, ice, and silty loam. Combining our dataset with another dataset from the literature, we evaluate both a Convolutional Neural Network (CNN) and the novel state space model (SSM)-based Mamba architecture on a TC task. We show that while CNN outperforms Mamba on each separate dataset, Mamba achieves greater accuracy when trained on a combination of both. In addition, we demonstrate that Mamba's learning capacity is greater than a CNN for increasing amounts of data. We show that the combination of two TC datasets yields a latent space that can be interpreted with the properties of the terrains. We also discuss the implications of merging datasets on classification. Our source code and dataset are publicly available online: https://github.com/norlab-ulaval/BorealTC.

Keywords: Field Robots, Mapping, Wheeled Robots
Abstract: Mobile robots operating in convoys have a limited view of the terrain to be traversed if it is occluded by the preceding vehicle. Furthermore, the preceding vehicle might change the terrain geometry and eventually significantly alter its traversability by driving over the terrain. When the following vehicles do not consider such changes, they can use spurious terrain appearance and geometry to decide whether to follow in the tracks of the previous vehicle or to avoid them since the preceding vehicle's tracks can make the terrain untraversable. We propose to predict the terrain changes induced by the robot traversal on the traversed terrain and thus support the decision-making of the following vehicles. The developed model projects the robot wheel footprint along the planned robot path and combines the projection with the terrain appearance and prior terrain elevation. The coupled model is used in a convolutional neural network that predicts the elevation after traversal. The footprint projection component is designed so that learned networks can be transferred to vehicles with different wheel footprints without relearning the model. The proposed model is verified using a dataset captured using a real, one-ton, six-wheel robot traversing rigid roads and vegetated fields.

Keywords: Field Robots, Motion and Path Planning, Reactive and Sensor-Based Planning
Abstract: In the context of unstructured and unknown environment, the autonomous navigation still faces many challenges, such as assessing rough terrain and deciding how to safely navigate complex terrain. In this work, we propose a robust and practical off-road navigation framework that has been successfully deployed on a vibroseis truck for land exploration. First, in degraded wild scenes, a tightly coupled lidar-GNSS-inertial fusion odometry and mapping framework is adopted to construct a local point cloud map around the vehicle in real-time and provide precise localization. Then, based on amplitude-frequency characteristic analysis and point cloud PCA, a multi-layer terrain assessment map containing terrain roughness, obstacles and slope information is obtained. Finally, combining Gaussian distribution based adaptive sampler and Bayesian sequentially updated proposal distribution, a local graph is efficiently built to obtain multiple path solutions under constrained conditions. Both simulations and field experiments show that the proposed navigation framework can decide how to travel on a flat road even in harsh terrain conditions, naturally suppressing frequent attitude angle changes and preventing vehicle accidents.

Keywords: Field Robots, Collision Avoidance, Motion and Path Planning
Abstract: Identifying traversable areas is a critical task for unmanned vehicles exploring safely through unstructured environments. In practice, the ambiguity in perceiving terrain traversability usually brings great challenges for autonomous exploration in unknown and uneven terrain, which often leads to conservative strategies or potential risk of vehicle damage, resulting in many unexplored areas in the environment. To that end, this paper proposes a plane-assisted autonomous robot exploration framework (PARE) to achieve maximum volume and safe autonomous exploration. The process is carried out by a three-step dual-layer framework: constructing a local tree using Plane-Assisted RRT* (PA-RRT*), calculating exploration gain based on terrain information, and maintaining a global search graph. Firstly, the planar feature metrics (flatness, sparsity, elevation variation, slope and slope variation) are introduced to determine the terrain traversability. Secondly, to completely explore the rugged environment, we propose a dual-layer exploration framework comprising local and global strategies. A local planner based on PA-RRT* is proposed to find the best path by evaluating the planar information and the volumetric gain within the local exploration tree. Meanwhile, a global planner constructed by graph is proposed to record unexplored nodes with high exploration gain from the local tree to ensure a high level of exploration volume. Extensive simulation and real-world experiments demonstrate that our method significantly outperforms existing frameworks, with an average improvement of more than 12% in exploration volume.

Keywords: Field Robots, Localization, Networked Robots
Abstract: With the goal of developing low-cost and innovative perception and localization techniques for autonomous vehicles, this work explores a system that solely relies on a LoRa receiver and a magnetometer for agent localization within urban environments. Using the received signal strength from LoRa beacons distributed across a test area of 16,000 square meters, a model of expected RSSI values per beacon is estimated using Gaussian Process (GP) regression. Motion is estimated using a probabilistic signal similarity classifier, and localization is obtained via a particle filter. Our experiments demonstrate that our proposed system is able to estimate our location to within three meters RMSE when the agent is within the convex hull of prior data. In real-world scenarios, characterized by signal interference and environmental complexities, our approach highlights the potential of leveraging affordable technology such as LoRa receivers and magnetometers for robust and accurate location estimation in complex urban environments. The integration of low-cost LoRa devices, Gaussian Process regression, particle filtering and our novel signal similarity motion estimator offers a promising avenue for achieving cost-effective localization solutions without compromising accuracy or reliability.

Keywords: Marine Robotics, Autonomous Vehicle Navigation, Mapping
Abstract: We propose and analyze a pipeline to transform raw sonar data from underwater vehicles into actionable information for Simultaneous Localization and Mapping (SLAM) in real time. The pipeline encompasses three sequential steps, each building upon state-of-the-art algorithms from the existing literature: swath processing to preprocess the raw sonar data, with a primary focus on eliminating blind zones and noise re- duction; transformation of these swaths into probabilistic maps of the surroundings of the sonar; and finally, detection and classification of underwater landmarks from the probabilistic maps. Our work contributes by modifying algorithms from the literature to ensure computational efficiency and integrating them so that they operate in sequence, thereby furnishing valuable information for navigation purposes. Through validation with field data, we then discuss which scenarios may prove difficult for the individual stages of the proposed pipeline. We provide indications on whether each step may encounter challenges and discuss how this occurrence may affect the overall quality of the final result. This empirical discussion is useful for discerning the practical applicability of the proposed pipeline in real-world settings.

Keywords: Wheeled Robots, Product Design, Development and Prototyping, Search and Rescue Robots
Abstract: In this study, a throwing robot equipped with an shock absorbing structure, utilizing paired-Cross Flexural Hinge (p-CFH) and an airbag, was fabricated and validated to assess the effectiveness of its impact absorption mechanism. This robot was developed in anticipation of situations where direct human intervention for life rescue would be challenging. Throwing robots can be broadly categorized into ball type, wheel type, and hybrid type. The hybrid type combines the advantages of both: the ease of throwing from ball type, due to its low air resistance coefficient, and the versatile mobility of the wheel type in diverse environments. However, hybrid type throwing robots are more vulnerable to external impacts due to the complexity of their internal structure, resulting in a lower maximum drop height compared to wheel type robots. To address these challenges, this research proposes a the Throwbot that combines the easy throwing capability of ball type with the obstacle overcoming ability of the wheel type, while also addressing the low free fall height drawback inherent in hybrid types. To achieve this, we developed a Throwbot with a ball to wheel transform structure, p-CFH mechanism, and airbag based impact absorption system. Additionally, materials were selected based on simulation results to refine the Throw- bot. The performance of the proposed robot was evaluated through various assessments, including free fall experiments and obstacle overcoming tests. Through this research, the proposed Throwbot effectively addresses the shortcomings of existing throwing robots, establishing a novel approach to throwing robot design.

Keywords: Agricultural Automation, Robotics and Automation in Agriculture and Forestry, Field Robots
Abstract: Cane pruning grapevines is a complex manual task requiring expert vine assessment to determine which canes to prune. This paper presents Archie Jnr, which was developed to autonomously assess the structure of the vine and prune the lower-quality canes as an expert pruner would. The platform has been extensively evaluated in a real-world commercial vineyard using a three-cane pruning method. The results show the effectiveness of the vision system for generating accurate assessments of a vine’s canes. The platform is also shown to be capable of successfully pruning 71.1% of the 311 total canes that required pruning across 25 vines.

Keywords: Service Robotics, Software-Hardware Integration for Robot Systems, Field Robots
Abstract: Culverts, essential components of drainage systems, require regular inspection to ensure optimal functionality. However, culvert inspections pose numerous challenges, including accessibility, manpower, defect localization, and reliance on superficial assessments. To address these challenges, we propose a novel Culvert Autonomous Inspection Robotic System (CAIS) equipped with advanced sensing and evaluation capabilities. Our solution integrates an RGBD camera, deep learning, lighting systems, and non-destructive evaluation (NDE) techniques to enable accurate and comprehensive condition assessments. We present a pioneering Partially Observable Markov Decision Process (POMDP) framework to resolve uncertainty in autonomous inspections, especially in confined and unstructured environments like culverts or tunnels. The framework outputs detailed 3D maps highlighting visual defects and NDE condition assessments, demonstrating consistent and reliable performance in both indoor and outdoor scenarios. Additionally, we provide an open-source implementation of our framework on GitHub, contributing to the advancement of autonomous inspection technology and fostering collaboration within the research community.

Keywords: Agricultural Automation, Robotics and Automation in Agriculture and Forestry, Computer Vision for Automation
Abstract: Fish detection in water-land transfer has significantly contributed to the fishery. However, manual fish detection in crowd-collaboration performs inefficiently and expensively, involving insufficient accuracy. To further enhance the water-land transfer efficiency, improve detection accuracy, and reduce labor costs, this work designs a new type of lightweight and plug-and-play edge intelligent vision system to automatically conduct fast fish detection with high-speed camera. Moreover, a novel similarity-aware vision Transformer for fast fish detection (FishViT) is proposed to onboard identify every single fish in a dense and similar group. Specifically, a novel similarity-aware multi-level encoder is developed to enhance multi-scale features in parallel, thereby yielding discriminative representations for varying-size fish. Additionally, a new soft-threshold attention mechanism is introduced, which not only effectively eliminates background noise from images but also accurately recognizes both the edge details and overall features of different similar fish. 85 challenging video sequences with high framerate and high-resolution are collected to establish a benchmark from real fish water-land transfer scenarios. Exhaustive evaluation conducted with this challenging benchmark has proved the robustness and effectiveness of FishViT with over 80 FPS. Real work scenario tests validate the practicality of the proposed method. The code and demo video are available at https://github.com/vision4robotics/FishViT.

Keywords: Engineering for Robotic Systems, Field Robots, Automation Technologies for Smart Cities
Abstract: Vision-assisted container loading of Rubber Tyred Gantry (RTG) cranes are facing two primary challenges. Firstly, the uncertainty inherent in Covolutional Neural Network (CNN) based detection hinders its direct application in the safety-critical operation of such heavy-duty machinery. Secondly, sensor calibration introduces additional complexities and errors into the system. However, existing studies have not adequately addressed these challenges. Motivated by this gap, this paper proposes an integrated approach for target detection and alignment control in container loading of RTG cranes. To ensure reliable target marker identification, a heuristic postprocessing algorithm is developed as a complement to CNN-based foreground segmentation, thereby ensuring safety during the container handling process. On this basis, a pixel-based control scheme is designed to align the container with the target markers, which eliminates the need for offline or online sensor calibrations. The proposed approach has been successfully implemented on a real RTG crane manufactured by Shanghai Zhenhua Heavy Industries Co., Ltd. (ZPMC) and validated at the Port of Ningbo, China. Experimental results demonstrate the superiority of the proposed approach over current manual operations in port industries, highlighting its potential for crane automation.

Keywords: Robotics and Automation in Agriculture and Forestry, Mapping, Field Robots
Abstract: Terrestrial laser scanning (TLS) is the standard technique used to create accurate point clouds for digital forest inventories. However, the measurement process is de- manding, requiring up to two days per hectare for data collection, significant data storage, as well as resource-heavy post-processing of 3D data. In this work, we present a real-time mapping and analysis system that enables online generation of forest inventories using mobile laser scanners that can be mounted e.g. on mobile robots. Given incrementally created and locally accurate submaps—data payloads—our approach extracts tree candidates using a custom, Voronoi-inspired clus- tering algorithm. Tree candidates are reconstructed using an algorithm based on the Hough transform, which enables robust modeling of the tree stem. Further, we explicitly incorporate the incremental nature of the data collection by consistently updating the database using a pose graph LiDAR SLAM system. This enables us to refine our estimates of the tree traits if an area is revisited later during a mission. We demonstrate competitive accuracy to TLS or manual measurements using laser scanners that we mounted on backpacks or mobile robots operating in conifer, broad-leaf and mixed forests. Our results achieve RMSE of 1.93 cm, a bias of 0.65 cm and a standard deviation of 1.81 cm (averaged across these sequences)—with no post-processing required after the mission is complete.

Keywords: Robotics and Automation in Construction, Field Robots, AI-Based Methods
Abstract: Robust moisture detection is crucial for building maintenance and cost reduction. Current methods are often limited by the type of roofing material or are cumbersome and expensive. Ground Penetrating Radar (GPR) has shown promise in recent works in moisture detection due to its effectiveness across a broader range of materials, its compactness and lightweight nature, and its ability to image the subsurface. We introduce Roofus, an integrated robotic moisture detection system for flat rooftops, designed to overcome traditional method limitations. It combines a remotely controlled robot with deep learning GPR data processing and automatic map generation. Real-world data is collected and manually annotated for supervised learning. We investigate a novel approach to interpreting GPR data via deep learning using Transformer-based classifiers. LiDAR inertial odometry is employed to integrate multiple individual GPR scans into a holistic moisture map over the rooftop. When evaluated against existing methods such as infrared thermal imaging, electrical capacitance surveys, and nuclear moisture gauges, our system shows promising viability for industry application.

Keywords: Robotics and Automation in Agriculture and Forestry, Field Robots
Abstract: Apple fruitlet thinning is critical in cultivating high-quality apples, requiring an expert workforce to manage the orchard. The thinning process requires precise mapping of fruitlet clusters across the tree branches to manage the desired load for each tree. This paper presents Archie Snr, which was developed to autonomously assess the current load of the tree and thin the excess apples as an expert thinner would. The platform has been extensively evaluated in a real-world commercial orchard. The results show the platform can generate an average load count accuracy of 82.1% with a recall of 93.3%. The system was then able to successfully thin 66.14% of the fruitlets from the canopy.

Keywords: Learning from Experience, Deep Learning in Grasping and Manipulation, Motion and Path Planning
Abstract: We integrate neural networks into kinodynamic motion planning and present the Learning for KinoDynamic Tree Expansion (L4KDE) method. Tree-based planning approaches, such as rapidly exploring random tree (RRT), are the dominant approach to finding globally optimal plans in continuous state-space motion planning. Central to these approaches is tree expansion, the procedure in which new nodes are added to an ever-expanding tree. We study the kinodynamic variants of tree-based planning, where we have known system dynamics and kinematic constraints. In the interest of quickly selecting nodes to connect newly sampled coordinates, existing methods typically cannot optimise the finding of nodes that have a low cost to transition to sampled coordinates. Instead, they use metrics like Euclidean distance between coordinates as a heuristic for selecting candidate nodes to connect to the search tree. We propose L4KDE to address this issue. L4KDE uses a neural network to predict transition costs between queried states, which can be efficiently computed in batch, providing much higher quality estimates of transition cost compared to commonly used heuristics while maintaining almost-surely asymptotic optimality guarantee. We empirically demonstrate the significant performance improvement provided by L4KDE on a variety of challenging system dynamics, with the ability to generalise across different instances of the same model class and in conjunction with a suite of modern tree-based motion planners.

Keywords: Machine Learning for Robot Control, Sensorimotor Learning, Bioinspired Robot Learning
Abstract: While different control approaches have been developed for smooth and safe navigation in service applications, they are limited by the needs for model-based assumptions, true training target/reward function, and/or large sample data. To overcome these limitations, this study proposes a model-free neural control architecture with a generic plug-and-play online Multiple Proactive Behavior Learning (MPL) module. The MPL adapts robot neural control policy in an online unsupervised manner with small sample data by correlating its sensory inputs to a local planner command. As a result, it allows a mobile robot to autonomously and quickly learn and balance various proactive behaviors related to smooth motion and collision avoidance. It also compensates for the limited planning update rates and the planning model mismatch of an arbitrary local motion planner. Compared with existing control approaches without the MPL, our control architecture with the MPL leads to (1) a 10% improvement in the smoothness of robot motion and 30% fewer collisions in a narrow static environment, and (2) trading motion smoothness for up to 70% fewer collisions in an unknown dynamic environment. Taken together, this study also demonstrates how to apply model-free neural control with unsupervised learning to existing model-based control (e.g., local motion planner) for efficient proactive behavior learning and control of mobile robots.

Keywords: Machine Learning for Robot Control, Model Learning for Control, Reinforcement Learning
Abstract: Reinforcement learning with domain randomization (DR) has been proposed as a promising approach for learning robust policies to environmental changes. However, for DR to work well in real-world environments, it is necessary to design appropriate DR distributions for model parameters. This paper proposes Normalizing Flow-based Parameter Distribution Estimation (NFPDE), a new estimation method for DR distributions. NFPDE models the target distribution by a flow-based generative model using normalizing flow and estimates the target distribution based on an offline dataset collected a priori in the target environment. Through numerical experiments on the OpenAI gym environment, we show that NFPDE can estimate the target distribution more accurately and efficiently than the previous estimation methods. We also show that the estimated DR distributions can improve the robustness of trained policies.

Keywords: Machine Learning for Robot Control, Aerial Systems: Applications
Abstract: In the last decade, data-driven approaches have become popular choices for quadrotor control, thanks to their ability to facilitate the adaptation to unknown or uncertain flight conditions. Among the different data-driven paradigms, Deep Reinforcement Learning (DRL) is currently one of the most explored. However, the design of DRL agents for Micro Aerial Vehicles (MAVs) remains an open challenge. While some works have studied the output configuration of these agents (i.e., what kind of control to compute), there is no general consensus on the type of input data these approaches should employ. Multiple works simply provide the DRL agent with full state information, without questioning if this might be redundant and unnecessarily complicate the learning process, or pose superfluous constraints on the availability of such information in real platforms. In this work, we provide an in-depth benchmark analysis of different configurations of the observation space. We optimize multiple DRL agents in simulated environments with different input choices and study their robustness and their sim-to-real transfer capabilities with zero-shot adaptation. We believe that the outcomes and discussions presented in this work supported by extensive experimental results could be an important milestone in guiding future research on the development of DRL agents for aerial robot tasks.

Keywords: Machine Learning for Robot Control, Path Planning for Multiple Mobile Robots or Agents, Computer Vision for Automation
Abstract: Virtual tube is a two-dimensional or three-dimensional strip or tubular area similar to RSFC (Relative Safe Flight Corridor), which can provide smooth, feasible, and safe space for UAV swarm in environments with dense obstacles. In order to address the problem that current virtual tube planning methods are mainly based on complex heuristic algorithm with consuming time complexity, we modify the model architecture by introducing generative adversarial network (GAN), and propose a Tube-GAN model. Tube-GAN takes the key point prompt image and obstacle environment image as inputs, and outputs the image of the virtual tube, which transforms the optimization problem into an image generation problem, leveraging the performance of computational efficiency for the construction of virtual tube. The experimental results demonstrate that the proposed Tube-GAN model can quickly generate virtual tube in random environments (less than 25ms), providing a new direction for the construction of virtual tube in real-time.

Keywords: Machine Learning for Robot Control, Neural and Fuzzy Control, Learning from Demonstration
Abstract: This paper introduces a new method for safety-aware robot learning, focusing on repairing policies using predictive models. Our method combines behavioral cloning with neural network repair in a two-step supervised learning framework. It first learns a policy from expert demonstrations and then applies repair subject to predictive models to enforce safety constraints. The predictive models can encompass various aspects relevant to robot learning applications, such as proprioceptive states and collision likelihood. Our experimental results demonstrate that the learned policy successfully adheres to a predefined set of safety constraints on two applications: mobile robot navigation, and real-world lower-leg prostheses. Additionally, we have shown that our method effectively reduces repeated interaction with the robot, leading to substantial time savings during the learning process.

Keywords: Machine Learning for Robot Control, Manipulation Planning, Motion and Path Planning
Abstract: Suction cups are popular for picking and transporting packages in warehouse applications. To maximize throughput, high transport speeds are desired. Many packages are deformable and may detach from the suction cups due to inertial loading if trajectories use excessive velocities. This paper introduces a novel methodology that analyzes package deformation through its curvature at the package-suction cup contact interface to generate a Factor-of-Safety (FOS) score for each waypoint in a given trajectory. By maintaining the FOS above a predetermined threshold, the trajectory planner is able to generate transport trajectories that are both safe and time-optimized. Experimental results show the method's efficacy, demonstrating a 21.92% reduction in transport times compared to a conservative trajectory generation. Our FOS predictor identified trajectories that ensured safe package transport with 100% accuracy across all 627 real-world experiments.

Keywords: Model Learning for Control
Abstract: Robotic Fish is increasingly employed in various aquatic application, where modelling and controlling their dynamic behaviour are crucial. Despite considerable efforts in robotic fish dynamic modeling, controllers based on existing models cannot perform satisfactorily in non-stationary flow environments. The main reason is that the nonlinear and two-way coupled nature of the interaction between background flow and robotic fish dynamics underscores a significant challenge in integrating the influence of background flow on robotic fish behavior, a topic that remains relatively under-explored in existing literature. To this end, we propose a novel approach for dynamic modeling of robotic fish considering background flow using Koopman operators, namely Flow-Aware Robot-Fish Modeling (FARM). Specifically, we first collect motion data of the robotic fish in different background flow fields, and then obtain a linear approximation (dynamic model) of nonlinear dynamics through carefully selected lifted functions. The obtained model can provide the next state at the given current state, control input, and average flow velocity of the background flow field. We evaluate the accuracy of the obtained model by evaluating the RMSE of predicted motion trajectories and real trajectories in different flow field environments. The results indicate that the FARM is highly promising in obtain reliable dynamic model, and obtained model can achieve comparable prediction accuracy even in unseen flow field environments with rough flow map for activity region. This lays a solid foundation for the motion control of robotic fish in different background flow fields.

Keywords: Model Learning for Control, Human-Centered Robotics, Wearable Robotics
Abstract: Ensuring safety and adapting to the user's behavior are of paramount importance in physical human-robot interaction. Thus, incorporating elastic actuators in the robot's mechanical design has become popular, since it offers intrinsic compliance and additionally provide a coarse estimate for the interaction force by measuring the deformation of the elastic components. While observer-based methods have been shown to improve these estimates, they rely on accurate models of the system, which are challenging to obtain in complex operating environments. In this work, we overcome this issue by learning the unknown dynamics components using Gaussian process (GP) regression. By employing the learned model in a Bayesian filtering framework, we improve the estimation accuracy and additionally obtain an observer that explicitly considers local model uncertainty in the confidence measure of the state estimate. Furthermore, we derive guaranteed estimation error bounds, thus, facilitating the use in safety-critical applications. We demonstrate the effectiveness of the proposed approach experimentally in a human-exoskeleton interaction scenario.

Keywords: Model Learning for Control, Reinforcement Learning, Machine Learning for Robot Control
Abstract: Model-free reinforcement learning (RL) is inher- ently a reactive method, operating under the assumption that it starts with no prior knowledge of the system and entirely depends on trial-and-error for learning. This approach faces several challenges, such as poor sample efficiency, generaliza- tion, and the need for well-designed reward functions to guide learning effectively. On the other hand, controllers based on complete system dynamics do not require data. This paper addresses the intermediate situation where there is not enough model information for complete controller design, but there is enough to suggest that a model-free approach is not the best approach either. By carefully decoupling known and unknown information about the system dynamics, we obtain an embedded controller guided by our partial model and thus improve the learning efficiency of an RL-enhanced approach. A modular design allows us to deploy mainstream RL algorithms to refine the policy. Simulation results show that our method signifi- cantly improves sample efficiency compared with standard RL methods on continuous control tasks, and also offers enhanced performance over traditional control approaches. Experiments on a real ground vehicle also validate the performance of our method, including generalization and robustness.

Keywords: Sensorimotor Learning, Legged Robots
Abstract: Quadruped robots have shown remarkable mobility on various terrains through reinforcement learning. Yet, in the presence of sparse footholds and risky terrains such as stepping stones and balance beams, which require precise foot placement to avoid falls, model-based approaches are often used. In this paper, we show that end-to-end reinforcement learning can also enable the robot to traverse risky terrains with dynamic motions. To this end, our approach involves training a generalist policy for agile locomotion on disorderly and sparse stepping stones before transferring its reusable knowledge to various more challenging terrains by finetuning specialist policies from it. Given that the robot needs to rapidly adapt its velocity on these terrains, we formulate the task as a navigation task instead of the commonly used velocity tracking which constrains the robot's behavior and propose an exploration strategy to overcome sparse rewards and achieve high robustness. We validate our proposed method through simulation and real-world experiments on an ANYmal-D robot achieving peak forward velocity of >= 2.5 m/s on sparse stepping stones and narrow balance beams. Video: youtu.be/Z5X0J8OH6z4

Keywords: Sensorimotor Learning, Imitation Learning, Learning from Experience
Abstract: In recent years, studies have been actively conducted on combining large language models (LLM) and robotics; however, most have not considered end-to-end feedback in the robot-motion generation phase. The prediction of deep neural networks must contain errors, it is required to update the trained model to correspond to the real environment to generate robot motion adaptively. This study proposes an integration method that connects the robot-motion learning model and LLM using shared latent variables. When generating robot motion, the proposed method updates shared parameters based on prediction errors from both sensorimotor attention points and task language instructions given to the robot. This allows the model to search for latent parameters appropriate for the robot task efficiently. Through simulator experiments on multiple robot tasks, we demonstrated the effectiveness of our proposed method from two perspectives: position generalization and language instruction generalization abilities.

Keywords: Perception-Action Coupling, Reinforcement Learning, Legged Robots
Abstract: Multi-task robot learning holds significant importance in tackling diverse and complex scenarios. However, current approaches are hindered by performance issues and difficulties in collecting training datasets. In this paper, we propose GeRM (Generalist Robotic Model). We utilize offline reinforcement learning to optimize data utilization strategies to learn from both demonstrations and sub-optimal data, thus surpassing the limitations of human demonstrations. Thereafter, we employ a transformer-based VLA network to process multi-modal inputs and output actions. By introducing the Mixture-of-Experts structure, GeRM allows faster inference speed with higher whole model capacity, and thus resolves the issue of limited RL parameters, enhancing model performance in multi-task learning while controlling computational costs. Through a series of experiments, we demonstrate that GeRM outperforms other methods across all tasks, while also validating its efficiency in both training and inference processes. Additionally, we uncover its potential to acquire emergent skills. Additionally, we contribute the QUARD-Auto dataset, collected automatically to support our training approach and foster advancements in multi-task quadruped robot learning. This work presents a new paradigm for reducing the cost of collecting robot data and driving progress in the multi-task learning community.

Keywords: Probability and Statistical Methods, Reinforcement Learning, Marine Robotics
Abstract: Due to the complexity of many decision making problems, tree search algorithms often have inadequate information to produce accurate transition models. This results in ambiguities (uncertainties for which there are multiple plausible models). Faced with ambiguities, robust methods have been used to produce safe solutions—often by maximizing the lower bound over the set of plausible transition models. However, they often overlook how much the representation of uncertainty can impact how a decision is made. This work introduces the Ambiguity Attitude Graph Search (AAGS), advocating for more comprehensive representations of ambiguities in decision making. Additionally, AAGS allows users to adjust their ambiguity attitude (or preference), promoting exploration and improving users’ ability to control how an agent should respond when faced with a set of plausible alternatives. Simulation in a dynamic sailing environment shows how environments with high entropy transition models can lead robust methods to fail. Results further demonstrate how adjusting ambiguity attitudes better fulfills objectives while mitigating this failure mode of robust approaches. Because this approach is a generalization of the robust framework, these results further demonstrate how algorithms focused on ambiguity have applicability beyond safety-critical systems.

Keywords: Evolutionary Robotics, Machine Learning for Robot Control, Learning from Experience
Abstract: Meta-Reinforcement Learning (Meta-RL) is a machine learning paradigm aimed at learning reinforcement learning policies that can quickly adapt to unseen tasks with few-shot data. Nevertheless, applying Meta-RL to real-world applications faces challenges due to the cost of data acquisition. To address this problem, offline Meta-RL has emerged as a promising solution, focusing on learning policies from pre-collected data that can effectively and rapidly adapt to unseen tasks. In this paper, we propose a new offline Meta-RL method called Meta-Actor-Critic with Evolving Gradient Agreement (MACEGA). MACEGA utilizes an evolutionary approach to estimate meta-gradients conductive to generalization across unseen tasks. During meta-training, gradient evolution is utilized to meta-update the value network and policies. Moreover, we use gradient agreement as an optimization objective for meta-learning, thereby enhancing the generalization ability of the meta-policy. We experimentally demonstrate the robustness of MACEGA in handling offline data quality. Furthermore, extensive experiments on various benchmarks provide empirical evidence that MACEGA outperforms previous state-of-the-art methods in generalizing to unseen tasks, thus demonstrating its potential for real-world applications.

Keywords: Learning from Experience, Probabilistic Inference, Imitation Learning
Abstract: Probabilistic Movement Primitives (ProMPs) and their variants are powerful methods for robots to learn complex tasks from human demonstrations, where motion trajectories are represented as stochastic processes with Gaussian assumptions. However, despite being computationally efficient, this method has limited expressivity in capturing the diversity found in human demonstrations, which are typically characterised by the multi- modality of motions. For example, when picking up an object partially obscured by an obstacle, some individuals may opt to go to the right while others may choose the left side of the object. In this paper, we introduce Stein Movement Primitives (SMPs), a novel approach to probabilistic movement primitives. We formulate motion primitive adaptation as non-parametric probabilistic inference using Stein Variational Gradient Descent (SVGD), thus not constraining our method to any posterior distribution assumption and enabling direct representation of the multi-modality in human demonstrations. We illustrate how our method can adapt robot motion to different scenarios for collision avoidance and adaptation to new tasks without being restricted to single modal assumptions. Experimentally, we demonstrate our approach on several domain adaptation problems using the LASA dataset and with a real robotic arm.

Keywords: Wheeled Robots, Representation Learning, Transfer Learning
Abstract: Model-based controllers using a linearized model around the system’s equilibrium point is a common approach in the control of a wheeled humanoid due to their less computational load and ease of stability analysis. However, controlling a wheeled humanoid robot while it lifts an unknown object presents significant challenges, primarily due to the lack of knowledge in object dynamics. This paper presents a framework designed for predicting the new equilibrium point explicitly to control a wheeled-legged robot with unknown dynamics.We estimated the total mass and center of mass of the system from its response to initially unknown dynamics, then calculated the new equilibrium point accordingly. To avoid using additional sensors (e.g., force torque sensor) and reduce the effort of obtaining expensive real data, a data-driven approach is utilized with a novel real-to-sim adaptation. A more accurate nonlinear dynamics model, offering a closer representation of real-world physics, is injected into a rigid-body simulation for real-to-sim adaptation. The nonlinear dynamics model parameters were optimized using Particle Swarm Optimization. The efficacy of this framework was validated on a physical wheeled inverted pendulum, a simplified model of a wheeledlegged robot. The experimental results indicate that employing a more precise analytical model with optimized parameters significantly reduces the gap between simulation and reality, thus improving the efficiency of a model-based controller in controlling a wheeled robot with unknown dynamics.

Keywords: Art and Entertainment Robotics, Optimization and Optimal Control, AI-Based Methods
Abstract: This paper introduces CLIPSwarm, a new algorithm designed to automate the modeling of swarm drone formations based on natural language. The algorithm begins by enriching a provided word, to compose a text prompt that serves as input to an iterative approach to find the formation that best matches the provided word. The algorithm iteratively refines formations of robots to align with the textual description, employing different steps for “exploration” and “exploitation”. Our framework is currently evaluated on simple formation targets, limited to con- tour shapes. A formation is visually represented through alpha-shape contours and the most representative color is automatically found for the input word. To measure the similarity between the description and the visual representation of the formation, we use CLIP [1], encoding text and images into vectors and assessing their similarity. Subsequently, the algorithm rearranges the formation to visually represent the word more effectively, within the given constraints of available drones. Control actions are then assigned to the drones, ensuring robotic behavior and collision-free movement. Experimental results demonstrate the system’s efficacy in accurately modeling robot formations from natural language descriptions. The algorithm’s versatility is showcased through the execution of drone shows in photorealistic simulation with varying shapes. We refer the reader to the supplementary video for a visual reference of the results.

Keywords: Computational Geometry, Optimization and Optimal Control, SLAM
Abstract: We present a novel two-view geometry estimation framework which is based on a differentiable robust loss function fitting. We propose to treat the robust fundamental matrix estimation as an implicit layer, which allows us to avoid backpropagation through time and significantly improves the numerical stability. To take full advantage of the information from the feature matching stage we incorporate learnable weights that depend on the matching confidences. In this way our solution brings together feature extraction, matching and two-view geometry estimation in a unified end-to-end trainable pipeline. We evaluate our approach on the camera pose estimation task in both outdoor and indoor scenarios. The experiments on several datasets show that the proposed method outperforms both classic and learning-based state-of-the-art methods by a large margin. The project webpage is available at: https://github.com/VladPyatov/ihls

Keywords: Optimization and Optimal Control, Multi-Contact Whole-Body Motion Planning and Control, Legged Robots
Abstract: This paper presents a stochastic/robust nonlinear model predictive control (NMPC) to enhance the robustness of model-based legged locomotion against contact uncertainties. We integrate the contact uncertainties into the covariance propagation of stochastic/robust NMPC framework by leveraging the guard saltation matrix and an extended Kalman filter-like covariance update. We achieve fast stochastic/robust NMPC computation by utilizing the zero-order algorithm with additional improvements in computational efficiency concerning the feedback gains. We conducted numerical experiments and demonstrate that the proposed method can accurately forecast future state covariance and generate trajectories that satisfies constraints even in the presence of the contact uncertainties. Hardware experiments on the perceptive locomotion of a wheeled-legged robot were also carried out, validating the feasibility of the proposed method in a real-world system with limited on-board computation.

Keywords: Optimization and Optimal Control, Legged Robots, Dynamics
Abstract: The current state-of-the-art gradient-based optimisation frameworks are able to produce impressive dynamic manoeuvres such as linear and rotational jumps. However, these methods, which optimise over the full rigid-body dynamics of the robot, often require precise foothold locations apriori, while real time performance is not guaranteed without elaborate regularisation and tuning of the cost function. In contrast, we investigate the advantages of a task-space optimisation framework, with special focus on acrobatic motions. Our proposed formulation exploits the system’s high-order nonlinearities, such as the nonholonomy of the angular momentum, in order to produce feasible, high acceleration manoeuvres. By leveraging the full-centroidal dynamics of the quadruped ANYmal C and directly optimising its footholds and contact forces, the framework is capable of producing efficient motion plans with low computational overhead. Finally, we deploy our proposed framework on the ANYmal C platform, and demonstrate its true capabilities through real-world experiments, with the successful execution of high-acceleration motions, such as linear and rotational jumps. Extensive analysis of these shows that the robot’s dynamics can be exploited to surpass its hardware limitations of having a high mass and low-torque limits.

Keywords: Optimization and Optimal Control, Kinematics, Embedded Systems for Robotic and Automation
Abstract: This work proposes an optimal control scheme for a trajectory-tracking Automated Guided Vehicle considering motion and collision constraints in a warehouse environment. We outline how the simpler obstacle avoidance constraints in the Cartesian Coordinate Frame (CCF) can be retained, while projecting the tricycle kinematics to the Frenet Coordinate Frame (FCF) for track progress. The Nonlinear Model Pre- dictive Control (NMPC) scheme is subsequently implemented using acados and its real-time feasibility is demonstrated in simulation and aboard a test vehicle at a warehouse.

Keywords: Optimization and Optimal Control, Robot Safety
Abstract: This paper proposes an optimization-based control framework for the torque-controlled robot, which can satisfy the joint position, velocity, and acceleration constraints under a bounded jerk. The optimization filter is incorporated as a module to modify the nominal controller output to ensure joint constraints. To formulate the optimization problem as a QP, the torque optimization problem is converted to the jerk optimization problem using the augmented state, and the constraints are reformulated to be affine in the jerk. Here, the viable constraints are derived using the time-optimal braking policy to guarantee the feasibility of the QP. The proposed method was validated in simulation and with a 6-DOF robot manipulator.

Keywords: Optimization and Optimal Control, Reinforcement Learning, Agent-Based Systems
Abstract: We investigate a pursuit-evasion game taking place in an unbounded three-dimensional space, where a flexible pursuer with hybrid dynamics collaborates with a fast pursuer and aims to capture a flexible evader within a finite time. The key feature of this problem lies in the hybrid dynamics of the flexible pursuer, which can change its dynamics once during the game and switch to a fast pursuer with increased speed but lower maneuverability. To address this challenge, we devise a hybrid strategy based on the soft actor-critic framework, tailored specifically for the flexible pursuer, which encompasses both maneuvering and switch tactics. We introduce a switch factor to the input of the actor network and incorporate switch actions to further expand the action space. These additions enable the flexible pursuer to execute maneuvering actions and determine a moment to switch to a fast pursuer. The reward function is designed to account for related angle, altitude, speed, and sparse reward. Through extensive ablation experiments conducted in a simulated environment, we demonstrate the efficacy of our algorithm in facilitating the learning of hybrid strategies for the flexible pursuer, resulting in significantly improved capture rates compared to alternative methods.

Keywords: Optimization and Optimal Control, Continual Learning, Learning from Experience
Abstract: This paper presents a novel experience-based technique, called EHGO, for sample-efficient adaptive control of nonlinear systems in the presence of dynamical modeling errors. The starting point for EHGO is a database seeded with many trajectories optimized under a reference estimate of real system dynamics. When executed on the real system, these trajectories will be suboptimal due to errors in the reference dynamics. The approach then leverages a hybrid gradient optimization technique, GRILC, which observes executed trajectories and computes gradients from the reference model to refine the control policy without requiring an explicit model of the real system. In past work, GRILC was applied in a restrictive setting in which a robot executes multiple rollouts from identical start states. In this paper, we show how to leverage a database to enable GRILC to operate across a wide envelope of possible start states in different iterations. The database is used to balance between start state proximity and recentness-of-experience via a learned distance metric to generate good initial guesses. Experiments on three dynamical systems (pendulum, car, drone) show that the proposed approach adapts quickly to online experience even when the reference model has significant errors. In these examples EHGO generates near-optimal solutions within hundreds of epochs of real execution, which can be orders of magnitude more sample efficient than reinforcement learning techniques.

Keywords: Optimization and Optimal Control, Motion and Path Planning, Autonomous Vehicle Navigation
Abstract: Navigation of wheeled vehicles on uneven terrain necessitates going beyond the 2D approaches for trajectory planning. Specifically, it is essential to incorporate the full 6dof variation of vehicle pose and its associated stability cost in the planning process. To this end, most recent works aim to learn a neural network model to predict vehicle evolution. However, such approaches are data-intensive and fraught with generalization issues.

In this paper, we present a purely model-based approach that just requires the digital elevation information of the terrain. Specifically, we express the wheel-terrain interaction and 6dof pose prediction as a non-linear least squares (NLS) problem. As a result, trajectory planning can be viewed as a bi-level optimization. The inner optimization layer predicts the pose on the terrain along a given trajectory, while the outer layer deforms the trajectory itself to reduce the stability and kinematic costs of the pose.

We improve the state-of-the-art in the following respects. First, we show that our NLS-based pose prediction closely matches the output of a high-fidelity physics engine. This result, coupled with the fact that we can query gradients of the NLS solver, makes our pose predictor a differentiable wheel-terrain interaction model. We further leverage this differentiability to efficiently solve the proposed bi-level trajectory optimization problem. Finally, we perform extensive experiments and comparisons with a baseline to showcase the effectiveness of our approach in obtaining smooth, stable trajectories.

Keywords: Optimization and Optimal Control, Motion Control
Abstract: While robust model predictive control (MPC) has been studied extensively in recent decades, addressing unmatched disturbances in underactuated robotic systems is still challenging. In this paper, we propose a method to enhance the robustness of the MPC through the online estimation of disturbances using a nonlinear disturbance observer (NDOB). We call this method disturbance-aware MPC (DA-MPC), because the proposed method explicitly utilizes the estimated disturbance in the future prediction. We provide a performance analysis of the NDOB, establishing the boundedness between the predicted and real states. The main advantages of the DA-MPC include its applicability to real-time control and its compatibility with off-the-shelf optimal control problem (OCP) solvers. We demonstrate the application of the proposed method using an underactuated quadrotor system. The simulation validation shows the effectiveness of the proposed method compared to L1-adaptive MPC, which is one of the state-of-the-art robust MPC methods.

Keywords: Motion Control, Optimization and Optimal Control, Legged Robots
Abstract: Exploring the limits of quadruped robot agility, particularly in the context of rapid and real-time planning and execution of omnidirectional jump trajectories, presents significant challenges due to the complex dynamics involved, especially when considering significant impulse contacts. This paper introduces a new framework to enable fast, omnidirectional jumping capabilities for quadruped robots. Utilizing minimum jerk technology, the proposed framework efficiently generates jump trajectories that exploit its analytical solutions, ensuring numerical stability and dynamic compatibility with minimal computational resources. The virtual model control is employed to formulate a Quadratic Programming (QP) optimization problem to accurately track the Center of Mass (CoM) trajectories during the jump phase. In contrast, whole-body control strategies facilitate precise and compliant landing motion. The framework's efficacy is demonstrated through its implementation on an enhanced version of the open-source textit{Mini Cheetah} robot. Omnidirectional jumps—including forward, backward, and other directional—were successfully executed, showcasing the robot's capability to perform rapid and consecutive jumps with an average trajectory generation and tracking solution time of merely 50 microseconds.

Keywords: Robust/Adaptive Control, Dynamics, Parallel Robots
Abstract: In this paper, we propose a novel adaptive feedforward super-twisting sliding mode control algorithm to resolve the tracking control problem of parallel manipulators. The proposed control scheme includes three main terms, (i) the standard super-twisting algorithm, (ii) an adaptive feedforward dynamic model, and (iii) a feedback term to ensure stability. The proposed controller provides robustness towards uncertainties and disturbances, less sensitive to measurement noise, and allows dynamic parameters adaptation of the manipulator while executing a certain task. Real-time experiments are conducted on a 3-DOF non-redundant Delta parallel robot, including two main scenarios, (i) nominal case, and (ii) robustness towards operating acceleration changes. The relevance of the proposed controller is verified experimentally in both scenarios and compared with two other controllers from the literature, including the standard and the feedforward super-twisting sliding mode control algorithms.

Keywords: Simulation and Animation, Optimization and Optimal Control
Abstract: Differentiable physics simulation provides an avenue to tackle previously intractable challenges through gradient-based optimization, thereby greatly improving the efficiency of solving robotics-related problems. To apply differentiable simulation in diverse robotic manipulation scenarios, a key challenge is to integrate various materials in a unified framework. We present SoftMAC, a differentiable simulation framework that couples soft bodies with articulated rigid bodies and clothes. SoftMAC simulates soft bodies with the continuum-mechanics-based Material Point Method (MPM). We provide a novel forecast-based contact model for MPM, which effectively reduces penetration without introducing other artifacts like unnatural rebound. To couple MPM particles with deformable and non-volumetric clothes meshes, we also propose a penetration tracing algorithm that reconstructs the signed distance field in local area. Diverging from previous works, SoftMAC simulates the complete dynamics of each modality and incorporates them into a cohesive system with an explicit and differentiable coupling mechanism. The feature empowers SoftMAC to handle a broader spectrum of interactions, such as soft bodies serving as manipulators and engaging with underactuated systems. We conducted comprehensive experiments to validate the effectiveness and accuracy of the proposed differentiable pipeline in downstream robotic manipulation applications. Supplementary materials are available on our project website at https://damianliumin.github.io/SoftMAC.

Keywords: Underactuated Robots, Mechanism Design, Optimization and Optimal Control
Abstract: Underactuated manipulators reduce the number of bulky motors, thereby enabling compact and mechanically robust designs. However, fewer actuators than joints means that the manipulator can only access a specific manifold within the joint space, which is particular to a given hardware configuration and can be low-dimensional and/or discontinuous. Determining an appropriate set of hardware parameters for this class of mechanisms, therefore, is difficult - even for traditional task-based co-optimization methods. In this paper, our goal is to implement a task-based design and policy co-optimization method for underactuated, tendon-driven manipulators. We first formulate a general model for an underactuated, tendon-driven transmission. We then use this model to co-optimize a three-link, two-actuator kinematic chain using reinforcement learning. We demonstrate that our optimized tendon transmission and control policy can be transferred reliably to physical hardware with real-world reaching experiments.

Keywords: Environment Monitoring and Management, Task Planning
Abstract: This paper addresses the spatio-temporal areas restoration problem: a robot, with limited battery life, deployed in a known environment, needs to persistently plan a schedule to visit areas of interest and charge its battery as needed. The goal is to restore the areas' temporal properties that decay over time—--such as air quality--—so that the time the measured property values are below a certain threshold is minimized. This problem is different from typical problems solved in the area of monitoring a spatio-temporal environment. A related problem is the orienteering problem, where a robot visits nodes to maximize the profit collected at each visited node within a time budget frame. That problem is NP-hard. The typical formulation considers static profit, while we consider a time-varying one. Given look-ahead time window or schedule length, we formulate the problem as an optimization search problem with a temporal objective, and devise a heuristic function that enables finding solutions in polynomial time. The heuristic evaluates the discounted opportunity costs of a visit—a concept borrowed from economics. We then develop a greedy algorithm that takes the immediate feasible visit that minimizes this heuristic. This strategy addresses a primary limitation of a recent approach in applications where being able to revisit highly urgent areas within the time window of the schedule is critical. We provide a theoretical analysis on lower and upper bounds for the problem. Extensive experimental results with a robotic simulator show that our method is able to keep the areas in the environment above the threshold better than other methods and closer to the optimal. This work can enable high-impact applications, such as environmental preservation.

Keywords: Motion and Path Planning, Task and Motion Planning, Path Planning for Multiple Mobile Robots or Agents
Abstract: Path planning has long been an important and active research area in robotics. To address challenges in high-dimensional motion planning, this study introduces the Force Direction Informed Trees (FDIT*), a sampling-based planner designed to enhance speed and cost-effectiveness in pathfinding. FDIT* builds upon the state-of-the-art informed sampling planner, the Effort Informed Trees (EIT*), by capitalizing on often-overlooked information in invalid vertices. It incorporates principles of physical force, particularly Coulomb's law. This approach proposes the elliptical k-nearest neighbors search method, enabling fast convergence navigation and avoiding high solution cost or infeasible paths by exploring more problem-specific search-worthy areas. It demonstrates benefits in search efficiency and cost reduction, particularly in confined, high-dimensional environments. It can be viewed as an extension of nearest neighbors search techniques. Fusing invalid vertex data with physical dynamics facilitates force-direction-based search regions, resulting in an improved convergence rate to the optimum. FDIT* outperforms existing single-query, sampling-based planners on the tested problems in R^4 to R^16 and has been demonstrated on a real-world mobile manipulation task.

Keywords: Semantic Scene Understanding, Task Planning
Abstract: Task planning for robots in real-life settings presents significant challenges. These challenges stem from three primary issues: the difficulty in identifying grounded sequences of steps to achieve a goal; the lack of a standardized mapping between high-level actions and low-level commands; and the challenge of maintaining low computational overhead given the limited resources of robotic hardware. We introduce EMPOWER, a framework designed for open-vocabulary online grounding and planning for embodied agents aimed at addressing these issues. By leveraging efficient pre-trained foundation models and a multi-role mechanism, EMPOWER demonstrates notable improvements in grounded planning and execution. Quantitative results highlight the effectiveness of our approach, achieving an average success rate of 0.73 across six different real-life scenarios using a TIAGo robot.

Keywords: Task and Motion Planning, Manipulation Planning, Service Robotics
Abstract: Integrated Task and Motion Planning (TAMP) offers opportunities for achieving generalized autonomy in robots but also poses challenges. It involves searching in both symbolic task space and high-dimensional motion space, while also addressing geometrically infeasible actions within its hierarchical process. We introduce a novel TAMP decision- making framework, utilizing an extended decision tree for both symbolic task planning and high-dimensional motion variable binding. Employing top-k planning, we generate a skeleton space with diverse candidate plans, seamlessly integrating it with motion variable spaces into an extended decision space. Subsequently, Monte-Carlo Tree Search (MCTS) is utilized to maintain a balance between exploration and exploitation at decision nodes, ultimately yielding optimal solutions. Our approach combines symbolic top-k planning with concrete mo- tion variable binding, leveraging MCTS for proven optimality, resulting in a powerful algorithm for handling combinatorial complexity in long-horizon manipulation tasks. Empirical eval- uations demonstrate the algorithm’s effectiveness in diverse, challenging robot tasks, in comparison with its most competitive baseline method.

Keywords: Task and Motion Planning, Search and Rescue Robots, Mapping
Abstract: Rapid sampling from the environment to acquire available frontier points and timely incorporating them into subsequent planning to reduce fragmented regions are critical to improve the efficiency of autonomous exploration. We propose HPHS, a fast and effective method for autonomous exploration of unknown environments. In this work, we quickly sample hybrid frontier points directly from the LiDAR data and the local map around the robot, and exploit hierarchical planning strategy to provide the robot with a global perspective. The hierarchical planning framework divides the environment into multiple subregions and arranges the order of access to them by evaluating the revenue of each subregion. The combination of the frontier point sampling method and hierarchical planning strategy reduces the complexity of the planning problem and mitigates the issue of region remnants during the exploration process. Detailed simulation and real world experiments demonstrate the effectiveness and efficiency of our approach in various aspects. The source code will be released to benefit the further research.

Keywords: Task and Motion Planning, Path Planning for Multiple Mobile Robots or Agents, Multi-Robot Systems
Abstract: This paper considers planning missions for a fleet of robots with limited energy. Each robot has size, heading, and velocity and its motion is described by non-linear differential equations. The dynamics of movements, existing obstacles, multiple robots, and waypoints are additional challenges, as the combined task and motion planning procedure prevents collisions. On their long-term missions, robots have to visit several waypoints in a cost-minimizing manner to satisfy the overall mission task. The robots consume energy and have to be recharged. The framework guides expanding a motion tree via a state projection to a discrete problem, whose solutions serve as search heuristics. Our experiments highlight that despite all these challenges, even sizable problem tasks can be solved even for complex environments

Keywords: Task and Motion Planning, AI-Based Methods, Continual Learning
Abstract: In dynamic open-world environments, agents continually face new challenges due to sudden and unpredictable novelties, hindering Task and Motion Planning (TAMP) in autonomous systems. We introduce a novel TAMP architecture that integrates symbolic planning with reinforcement learning to enable autonomous adaptation in such environments, operating without human guidance. Our approach employs symbolic goal representation within a goal-oriented learning framework, coupled with planner-guided goal identification, effectively managing abrupt changes where traditional reinforcement learning, re-planning, and hybrid methods fall short. Through sequential novelty injections in our experiments, we assess our method's adaptability to continual learning scenarios. Extensive simulations conducted in a robotics domain corroborate the superiority of our approach, demonstrating faster convergence to higher performance compared to traditional methods. The success of our framework in navigating diverse novelty scenarios within a continuous domain underscores its potential for critical real-world applications.

Keywords: Task and Motion Planning, Task Planning, AI-Based Methods
Abstract: Non-monotone object rearrangement planning in confined spaces such as cabinets and shelves is a widely occurring but challenging problem in robotics. Both the robot motion and the available regions for object relocation are highly constrained because of the limited space. This work proposes a Multi-Stage Monte Carlo Tree Search (MS-MCTS) method to solve non-monotone object rearrangement planning problems in confined spaces. Our approach decouples the complex problem into simpler subproblems using an object stage topology. A subgoal-focused tree expansion algorithm that jointly considers the high-level planning and the low-level robot motion is designed to reduce the search space and better guide the search process. By fitting the task into the MCTS paradigm, our method generates short object rearrangement sequences by balancing exploration and exploitation. The experiments demonstrate that our method outperforms the existing methods in terms of the planning time, the number of steps, the object moving distance and the gripper moving distance. Moreover, we deploy our MS-MCTS to a real-world robot system and verify its performance in different scenarios.

Keywords: Task and Motion Planning
Abstract: Conventional Task and Motion Planning (TAMP)approaches rely on manually designed interfaces connecting symbolic task planning with continuous motion generation. These domain-specific and labor-intensive modules are limited in addressing emerging tasks in real-world settings. Here, we present LLM^3, a novel LLM-based TAMP framework featuring a domain-independent interface. Specifically, we leverage the powerful reasoning and planning capabilities of pre-trained LLM to propose symbolic action sequences and select continuous action parameters for motion planning. Crucially, LLM^3 incorporates motion planning feedback through prompting, allowing the LLM to iteratively refine its proposals by reasoning about motion failure. Consequently, LLM^3 interfaces between task planning and motion planning, alleviating the intricate design process of handling domain-specific messages between them. Through a series of simulations in a box-packing domain, we quantitatively demonstrate the effectiveness of LLM^3 in solving TAMP problems and the efficiency in selecting action parameters. Ablation studies underscore the significant contribution of motion failure reasoning to the success of LLM^3. Furthermore, we conduct qualitative experiments on a physical manipulator, demonstrating the practical applicability of our approach in real-world settings.

Keywords: Task and Motion Planning, Vision-Based Navigation, Autonomous Agents
Abstract: Embodied navigation requires robots to understand and interact with the environment based on given tasks. Vision-Language Navigation (VLN) is an embodied navigation task, where a robot navigates within a previously seen and unseen environment, based on linguistic instruction and visual inputs. VLN agents need access to both local and global action spaces; former for immediate decision making and the latter for recovering from navigational mistakes. Prior VLN agents rely only on instruction-viewpoint alignment for local and global decision making and back-track to a previously visited viewpoint, if the instruction and its current viewpoint mismatches. These methods are prone to mistakes, due to the complexity of the instruction and partial observability of the environment. We posit that, back-tracking is sub-optimal and agent that is aware of its mistakes can recover efficiently. For optimal recovery, exploration should be extended to unexplored viewpoints (or frontiers). The optimal frontier is a recently observed but unexplored viewpoint that aligns with the instruction and is novel. We introduce a memory-based and mistake-aware path planning strategy for VLN agents, called StratXplore, that presents global and local action planning to select the optimal frontier for path correction. The proposed method collects all past actions and viewpoint features during navigation and then selects the optimal frontier suitable for recovery. Experimental results show this simple yet effective strategy improves the success rate on two VLN datasets with different task complexities.

Keywords: Task and Motion Planning, Manipulation Planning, Task Planning
Abstract: When multiple tablewares are closely packed on a table, rearranging obstacles to make space is necessary to grasp the target, often called target singulation. Due to the nature of handling fragile tablewares (i.e. plates, bowls), we make a few assumptions for the target singulation problem. First, tableware is grasped with a multi-fingered gripper; second, rearrangement is based on prehensile motions like pick-and- place. Under these assumptions, we aim to generate a relocation plan that guarantees global optimality. Furthermore, if any relocation plan cannot singulate the target, we aim to determine it quickly. Therefore, we propose a search method that utilizes the relationship between the object and its nearby obstacles expressed in propositional logic. We define the problem as determining logical entailment (i.e., whether the target can be singulated) and expand the search tree from the target while generating an optimal relocation plan. We demonstrate the performance of our algorithm by increasing the number of objects and validate the plan in a simulation environment.

Keywords: Task and Motion Planning, Physical Human-Robot Interaction, Formal Methods in Robotics and Automation
Abstract: Robots interacting with humans must be safe, reactive and adapt online to unforeseen environmental and task changes. Achieving these requirements concurrently is a challenge as interactive planners lack formal safety guarantees, while safe motion planners lack flexibility to adapt. To tackle this, we propose a modular control architecture that generates both safe and reactive motion plans for human-robot interaction by integrating temporal logic-based discrete task level plans with continuous Dynamical System (DS)-based motion plans. We formulate a reactive temporal logic formula that enables users to define task specifications through structured language, and propose a planning algorithm at the task level that generates a sequence of desired robot behaviors while being adaptive to environmental changes. At the motion level, we incorporate control Lyapunov functions and control barrier functions to compute stable and safe continuous motion plans for two types of robot behaviors: (i) complex, possibly periodic motions given by autonomous DS and (ii) time-critical tasks specified by Signal Temporal Logic~(STL). Our methodology is demonstrated on the Franka robot arm performing wiping tasks on a whiteboard and a mannequin that is compliant to human interactions and adaptive to environmental changes.

Keywords: Task Planning, Path Planning for Multiple Mobile Robots or Agents, Cooperating Robots
Abstract: In this work, we propose a novel method, NLNS-MASPF, to solve the Multi-Agent Scheduling and Pathfinding (MASPF) problem. The problem exhibits a bi-level structure, consisting of High-level Scheduling and Low-level Pathfinding. Our method applies a graph neural network in the high-level scheduling process and utilizes a MAPF solver with a schedule segmenting technique in the low-level pathfinding process. Through these approaches, NLNS-MASPF has experimentally demonstrated superior performance compared to the previous state-of-the-art MASPF algorithm, LNS-PBS, in solving the MASPF problem.

Keywords: Task Planning, Machine Learning for Robot Control, Manipulation Planning
Abstract: Large language models (LLMs) encode a vast amount of semantic knowledge and possess remarkable understanding and reasoning capabilities. Previous work has explored how to ground LLMs in robotic tasks to generate feasible and executable textual plans. However, low-level execution in the physical world may deviate from the high-level textual plan due to environmental perturbations or imperfect controller design. In this paper, we propose textbf{DoReMi}, a novel language model grounding framework that enables immediate Detection and Recovery from Misalignments between plan and execution. Specifically, we leverage LLMs to play a dual role, aiding not only in high-level planning but also generating constraints that can indicate misalignment during execution. Then vision language models (VLMs) are utilized to detect constraint violations continuously. Our pipeline can monitor the low-level execution and enable timely recovery if certain plan-execution misalignment occurs. Experiments on various complex tasks including robot arms and humanoid robots demonstrate that our method can lead to higher task success rates and shorter task completion times.

Keywords: Task Planning
Abstract: In this work, we introduce a strategy that frames the sequential action selection problem for robots in terms of resolving blocking conditions, i.e., situations that impede progress on an action en route to a goal. This strategy allows a robot to make one-at-a-time decisions that take in pertinent contextual information and swiftly adapt and react to current situations. We present a first instantiation of this strategy that combines a state-transition graph and a zero-shot Large Language Model (LLM). The state-transition graph tracks which previously attempted actions are currently blocked and which candidate actions may resolve existing blocking conditions. This information from the state-transition graph is used to automatically generate a prompt for the LLM, which then uses the given context and set of possible actions to select a single action to try next. This selection process is iterative, with each chosen and executed action further refining the state-transition graph, continuing until the agent either fulfills the goal criteria or encounters a termination condition. We demonstrate the effectiveness of our approach by comparing it to various LLM and traditional task-planning methods in a testbed of simulation experiments. We discuss the implications of our work based on our results.

Keywords: Task Planning, Multi-Robot Systems
Abstract: In this work, we introduce SMART-LLM, an innovative framework designed for embodied multi-robot task planning. SMART-LLM: Smart Multi-Agent Robot Task Planning using Large Language Models (LLMs), harnesses the power of LLMs to convert high-level task instructions provided as input into a multi-robot task plan. It accomplishes this by executing a series of stages, including task decomposition, coalition formation, and task allocation, all guided by programmatic LLM prompts within the few-shot prompting paradigm. We create a benchmark dataset designed for validating the multi-robot task planning problem, encompassing four distinct categories of high-level instructions that vary in task complexity. Our evaluation experiments span both simulation and real-world scenarios, demonstrating that the proposed model can achieve promising results for generating multi-robot task plans. The experimental videos, code, and datasets from the work can be found at https://sites.google.com/view/smart-llm/.

Keywords: Telerobotics and Teleoperation, Imitation Learning, Deep Learning Methods
Abstract: One of the critical aspects of assistive robotics is to provide a control system of a high-dimensional robot from a low-dimensional user input (i.e. a 2D joystick). Data-driven teleoperation seeks to provide an intuitive user interface called an action map to map the low dimensional input to robot velocities from human demonstrations. Action maps are machine learning models trained on robotic demonstration data to map user input directly to desired movements as opposed to aspects of robot pose (“move to cup or pour content” vs. “move along x- or y-axis”). Many works have investigated nonlinear action maps with multi-layer perceptrons, but recent work suggests that local-linear neural approximations provide better control of the system. However, local linear models assume actions exist on a linear subspace and may not capture nuanced motions in training data. In this work, we hypothesize that local-linear neural networks are effective because they make the action map odd w.r.t. the user input, enhancing the intuitiveness of the controller. Based on this assumption, we propose two nonlinear means of encoding odd behavior that do not constrain the action map to a local linear function. However, our analysis reveals that these models effectively behave like local linear models for relevant mappings between user joysticks and robot movements. We support this claim in simulation, and show on a realworld use case that there is no statistical benefit of using non-linear maps, according to the users experience. These negative results suggest that further investigation into model architectures beyond local linear models may offer diminishing returns for improving user experience in data-driven teleoperation systems.

Keywords: Telerobotics and Teleoperation, Learning from Demonstration, Bimanual Manipulation
Abstract: Humans can teleoperate robots to accomplish complex manipulation tasks. Imitation learning has emerged as a powerful framework that leverages human teleoperated demonstrations to teach robots new skills. However, the performance of the learned policies is bottlenecked by the quality, scale, and variety of the demonstration data. In this paper, we aim to lower the barrier to collecting large and high-quality human demonstration data by proposing a GEneraL framework for building LOw-cost and intuitive teleoperation systems for robotic manipulation (GELLO). Given a target robot arm, we build a GELLO controller device that has the same kinematic structure as the target arm, leveraging 3D-printed parts and economical off-the-shelf motors. GELLO is easy to build and intuitive to use. Through an extensive user study, we show that GELLO enables more reliable and efficient demonstration collection compared to other cost efficient teleoperation devices commonly used in the imitation learning literature such as virtual reality controllers and 3D spacemouses. We further demonstrate the capabilities of GELLO for performing complex bi-manual and contact-rich manipulation tasks. To make GELLO accessible to everyone, we have designed and built GELLO systems for 3 commonly used robotic arms: Franka, UR5, and xArm. All software and hardware are open-sourced and can be found on our website: https://wuphilipp.github.io/gello/.

Keywords: Telerobotics and Teleoperation, Dexterous Manipulation, Reinforcement Learning
Abstract: Dexterous telemanipulation is crucial in advancing human-robot systems, especially in tasks requiring precise and safe manipulation. However, it faces significant challenges due to the physical differences between human and robotic hands, the dynamic interaction with objects, and the indirect control and perception of the remote environment. Current approaches predominantly focus on mapping the human hand onto robotic counterparts to replicate motions, which exhibits a critical oversight: it often neglects the physical interaction with objects and relegates the interaction burden to the human to adapt and make laborious adjustments in response to the indirect and counter-intuitive observation of the remote environment. This work develops an End-Effects-Oriented Learning-based Dexterous Telemanipulation (EFOLD) framework to address telemanipulation tasks. EFOLD models telemanipulation as a Markov Game, introducing multiple end-effect features to interpret the human operator's commands during interaction with objects. These features are used by a Deep Reinforcement Learning policy to control the robot and reproduce such end effects. EFOLD was evaluated with real human subjects and two end-effect extraction methods for controlling a virtual Shadow Robot Hand in telemanipulation tasks. EFOLD achieved real-time control capability with low command following latency (delay<0.11s) and highly accurate tracking (MSE<0.084 rad).

Keywords: Telerobotics and Teleoperation, Haptics and Haptic Interfaces, Humanoid Robot Systems
Abstract: Teleoperating bipedal humanoid robots presents unique challenges, including decreased stability and reduced operator presence. This paper addresses these challenges by proposing a method that leverages the operator's inherent sense of stability by feedback from a sole haptics display to operate a bipedal humanoid robot. We developed a bilateral control system that integrates a device replicating sole haptics feedback and provides the operator with feedback on changes in the robot’s center of gravity. We conducted operating experiments in the forward-backward direction to evaluate its effectiveness and investigate the effectiveness of sole haptics on robot operation. The results demonstrate that operating with both vision and sole haptics feedback significantly reduces the robot's fall rate by over 56% when disturbances are applied, compared to using only vision feedback. Moreover, operators reported a 21% higher sense of presence with both vision and sole haptics feedback compared to using only vision feedback.

Keywords: Telerobotics and Teleoperation, Virtual Reality and Interfaces, Simulation and Animation
Abstract: This paper introduces the TactiMesh Teleoperator Interface (TTI), a novel predictive visual and haptic system designed explicitly for human-in-the-loop robot control using a head-mounted display (HMD). By employing simultaneous localization and mapping (SLAM) in tandem with a space carving method (CARV), TTI creates a real-time 3D surface mesh of remote environments from an RGB camera mounted on a Barrett WAM arm. The generated mesh is integrated into a physics simulator, featuring a digital twin of the WAM robot arm to create a virtual environment. In this virtual environment, TTI provides haptic feedback directly in response to the operator’s movements, eliminating the problem with delayed response from the haptic follower robot. Furthermore, texturing the 3D mesh with keyframes from SLAM allows the operator to control the viewpoint of their Head Mounted Dis- play (HMD) independently of the arm-mounted robot camera, giving a better visual immersion and improving manipulation speed. Incorporating predictive visual and haptic feedback significantly improves tele-operation in applications such as search and rescue, inspection, and remote maintenance.

Keywords: Telerobotics and Teleoperation, Physical Human-Robot Interaction, Safety in HRI
Abstract: Telemanipulation enables humans to perform tasks in dangerous environments without exposing them to any risk. The COVID-19 pandemic sadly showed, that these environments can also include the treatment and interaction with infected patients. Since human-robot interactions demand for low interaction forces yet high precision, telemanipulation often results in a high mental workload for the operator. To overcome this, we present a virtual guidance approach to perform telemanipulated insertion tasks. A nasopharyngeal swap sampling procedure is taken as use case. We extend our previously presented approach by adding an additional position fixture, introducing distance-dependent variable stiffness values and guaranteeing stability using energy tanks. Based on RGB-D data, the operator is guided towards a desirable insertion line while approaching the nostril. The distance-dependent stiffness values increase the smoothness of the fixture. Since variable stiffness values can result in unstable behavior, energy tanks for the fixtures are introduced. Experiments show the improvements compared to our previous approach. Further, a comparison between guided and unguided samplings performed by an expert user gives a first impression of the improvements resulting from the fixture.

Keywords: Telerobotics and Teleoperation, Object Detection, Segmentation and Categorization, Machine Learning for Robot Control
Abstract: Before the world-wide deployment of autonomous vehicles, it is essential to implement intermediate solutions with partial autonomy. One such solution is the use of vehicle teleoperation, the act of controlling a vehicle from a distance. In real time applications of teleoperation, it is often pertinent to use augmented reality components within the teleoperator view, which are referred to as a predictive display. In this work, we evaluate our predictive display method, which is a guiding path based on the free space in the environment. The path is generated based on our Dual Transformer Network (DTNet), which uses both object detection and lane semantic segmentation to define the free space in the environment. While the model has previously performed well on image data, it is necessary to observe its accuracy in the presence of time delay and packet loss, to assess its performance in a real-time setting. Thus, in this work, we use CARLA simulator to compare the detected free space on the teleoperator side to the true free space on the vehicle side across different values of time delay and packet loss. Under optimal network conditions, our model yielded a remarkable 87.9% DSC score and 81.3% IoU score. Defining our minimum performance threshold as 80% DSC and 70% IoU, we conclude that our model can effectively mitigate the challenges of time delay below 100ms and packet loss below 1%, both of which represent substantial tolerances.

Keywords: Telerobotics and Teleoperation, Human Factors and Human-in-the-Loop, Virtual Reality and Interfaces
Abstract: Shared control can ease and enhance a human operator's ability to teleoperate robots, particularly for intricate tasks demanding fine control over multiple degrees of freedom. However, the arbitration process dictating how much autonomous assistance to administer in shared control can confuse novice operators and impede their understanding of the robot's behavior. To overcome these adverse side-effects, we propose a novel formulation of shared control that enables operators to tailor the arbitration to their unique capabilities and preferences. Unlike prior approaches to customizable shared control where users could indirectly modify the latent parameters of the arbitration function by issuing a feedback command, we instead make these parameters observable and directly editable via a virtual reality (VR) interface. We present our user-customizable shared control method for a teleoperation task in SE(3), known as the buzz wire game. A user study is conducted with participants teleoperating a robotic arm in VR to complete the game. The experiment spanned two weeks per subject to investigate longitudinal trends. Our findings reveal that users allowed to interactively tune the arbitration parameters across trials generalize well to adaptations in the task, exhibiting improvements in precision and fluency over direct teleoperation and conventional shared control.

Keywords: Telerobotics and Teleoperation, Human Factors and Human-in-the-Loop, Design and Human Factors
Abstract: Teleoperated robots, especially in hazardous environments, integrate human cognition with machine efficiency, but can increase cognitive load, causing stress and reducing task performance and safety. This study examines the impact of the information available to the operator on cognitive load, physiological responses (e.g., GSR, blinking, facial temperature), and performance during teleoperation in three conditions: C1 - in presence, C2 - remote with Visual feedback, and C3 - remote with telepresence robot. The findings from our user study involving 20 participants show that information availability significantly impacts perceived cognitive load, as evidenced by the differences observed between conditions in our analysis. Furthermore, the results indicated that blinking rates varied significantly among the conditions. The results also underline that individuals with higher error scores on the spatial orientation test (SOT), reflecting lower spatial ability, are more likely to experience failure in conditions 2 and 3. The results show that information availability significantly affects cognitive load and teleoperation performance, especially depth perception of the robot's actions. Additionally, the thermal and GSR data findings indicate an increase in stress and anxiety levels when operators perform conditions 2 and 3, thus corroborating an increase in the user's cognitive load.

Keywords: Telerobotics and Teleoperation, Wheeled Robots, Space Robotics and Automation
Abstract: The difficulties posed by terrain-induced slippage for wheeled rovers traversing soft terrains are critical to ensuring safe and precise mobility. While bilateral teleoperation systems offer a promising solution to this issue, the inherent network-induced delays hinder the fidelity of the closed-loop integration, potentially compromising teleoperator system controls, and resulting in poor command-tracking performance. This work introduces a new model-free predictor framework based on deep learning designed to improve prediction performance and effectively compensate for large network delays in teleoperated wheeled rovers. Our approach employs the Recurrent Neural Network (RNN) to achieve a significant improvement in modeling complexity and prediction accuracy. Particularly, our framework consists of two distinct predictors, each tailored to the forward and backward coupling variables of the teleoperated wheeled rover. Human-in-the-loop experiments were conducted to validate the effectiveness of the developed framework in compensating for the delays encountered by teleoperated wheeled rovers coupled with terrain-induced slippage. The results confirm the improved prediction accuracy of the framework. This improvement is evidenced by improved performance and transparency metrics, which lead to better command-tracking performance.

Keywords: Telerobotics and Teleoperation, Motion Control, Human-Centered Robotics
Abstract: Performing human-like motion is crucial for service humanoid robots. Real-time motion retargeting allows clear observation of the robot's pose and provides instant feedback during human demonstrator actions. This paper presents an optimization-based real-time anthropomorphic motion retargeting framework for transferring human arm motion to a robot arm. The framework is generic, applicable to both spherical-rotational-spherical (SRS) and non-SRS robot arms. We introduce the normalized normal vector of the arm plane as an anthropomorphic criterion within our framework. The method is validated on a service humanoid robot, with both static and dynamic evaluations. The statistical analysis show that our method maintains strong anthropomorphic features while ensuring accurate wrist pose tracking.

Keywords: Telerobotics and Teleoperation, Haptics and Haptic Interfaces, Soft Sensors and Actuators
Abstract: This work describes the development, demonstration, and performance evaluation study of a wearable human machine interface (HMI) for robotic teleoperation. We present a novel tetherless HMI in the form of a backpack, body-worn 3-D arm motion capture sensors, finger flexion sensors, and haptic feedback muscles embedded in a glove. The system is integrated in a complete teleoperation framework, enabling users to be immersed in a remote environment through virtual reality headgear, facilitating intuitive manipulation of an industrial articulated arm. The designed HMI measures the kinematic configuration of the user’s arm, hand and fingers using multiple inertial measurement units (IMUs) and capacitive sensors respectively. Subsequently, the captured data is channeled into the teleoperation software stack. The gripping forces experienced at the robot’s end-effector are acquired using a custom three-dimensional Hall-effect magnetic sensor. The system simultaneously renders the kinesthetic and tactile feedback on the user’s fingers through custom designed pneumatically actuated soft robotic haptic muscles. The efficacy of the HMI and the teleoperation system was tested and evaluated by conducting user study experiments, which showed 31.4% faster teleoperation vs. a keypad controller and 60% less gripping force exerted when the haptics were enabled. The findings of the pilot study guided the design and prototype development of a printed electronics based stretchable sleeve and glove motion capture unit to improve the portability, ergonomics and user experience of the HMI.

Keywords: Localization, SLAM
Abstract: Prior structural reference like floor plan is readily accessible in indoor scene, which exhibits the potential of improving localization quality without the requirements of a previously-built high-precision map. This paper introduces a novel optimal transport-based framework for prior structural reference-based localization, aiming to improve the robustness for the robot localization. Leveraging the spacial relations of structures, a matching method based on optimal transport theory is proposed and it improves the robustness of matching results in dynamic scene and rapid rotation conditions. Additionally, this paper handles metric inaccuracies in the known structural reference by implementing an prior guided plane adjustment-based updating strategy. This strategy combines prior and observational information to jointly optimize the structural information within a sliding window. The performance of the framework is validated through real-world experiments, demonstrating superior accuracy and robustness to disturbances from dynamic occlusion and rapid rotation compared to common state-of-the-art SLAM and localization methods.

Keywords: Localization, Optimization and Optimal Control, Reactive and Sensor-Based Planning
Abstract: To enhance accuracy of robot state estimation, active sensing (or perception-aware) methods seek trajectories that maximize the information gathered by the sensors. To this aim, one possibility is to seek trajectories that minimize the (estimation error) covariance matrix output by an extended Kalman filter (EKF), w.r.t. its control inputs over a given horizon. However, this is computationally demanding. In this article, we derive novel backpropagation analytical formulas for the derivatives of the covariance matrices of an EKF w.r.t. all its inputs. We then leverage the obtained analytical gradients as an enabling technology to derive perception-aware optimal motion plans. Simulations validate the approach, showcasing improvements in execution time, notably over PyTorch's automatic differentiation. Experimental results on a real vehicle also support the method.

Keywords: Localization, Sensor Fusion
Abstract: Accurate and reliable ego-localization is critical for autonomous driving. In this paper, we present EgoVM, an end-to-end localization network that achieves comparable localization accuracy to prior state-of-the-art methods, but uses lightweight vectorized maps instead of heavy point-based maps. To begin with, we extract BEV features from online multi-view images and LiDAR point cloud. Then, we employ a set of learnable semantic embeddings to encode the semantic types of map elements and supervise them with semantic segmentation, to make their feature representation consistent with BEV features. After that, we feed map queries, composed of learnable semantic embeddings and coordinates of map elements, into a transformer decoder to perform cross-modality matching with BEV features. Finally, we adopt a robust histogram-based pose solver to estimate the optimal pose by searching exhaustively over candidate poses. We comprehensively validate the effectiveness of our method using both the nuScenes dataset and a newly collected dataset. The experimental results show that our method achieves centimeter-level localization accuracy, and outperforms existing methods using vectorized maps by a large margin. Furthermore, our model has been extensively tested in a large fleet of autonomous vehicles under various challenging urban scenes.

Keywords: Wheeled Robots, Sensor Fusion, Localization
Abstract: In mobile robotics, particularly in autonomous driving, localization is one of the key challenges for navigation and planning. For safe operation in the open world where vulnerable participants are present, precise and guaranteed safe localization is required. While current classical fusion approaches are safe due to provably bounded closed-form formulation, their situation-adaptivity is limited. In contrast, data-driven approaches are situation-adaptive based on the underlying training data but unbounded and unsafe. In our work, we propose a novel data-driven but provably bounded sensor fusion and apply it to mobile robotic localization. In extensive experiments using an autonomous driving test vehicle, we show that our fusion method outperforms other safe fusion approaches.

Keywords: Mapping, SLAM
Abstract: Recently the dense Simultaneous Localization and Mapping (SLAM) based on neural implicit representation has shown impressive progress in hole filling and high-fidelity mapping. Nevertheless, existing methods either heavily rely on known scene bounds or suffer inconsistent reconstruction due to drift in potential loop-closure regions, or both, which can be attributed to the inflexible representation and lack of local constraints. In this paper, we present LCP-Fusion, a neural implicit SLAM system with enhanced local constraints and computable prior, which takes the sparse voxel octree structure containing feature grids and SDF priors as hybrid scene representation, enabling the scalability and robustness during mapping and tracking. To enhance the local constraints, we propose a novel sliding window selection strategy based on visual overlap to address the loop-closure, and a practical warping loss to constrain relative poses. Moreover, we estimate SDF priors as coarse initialization for implicit features, which brings additional explicit constraints and robustness, especially when a light but efficient adaptive early ending is adopted. Experiments demonstrate that our method achieve better localization accuracy and reconstruction consistency than existing RGB-D implicit SLAM, especially in challenging real scenes (ScanNet) as well as self-captured scenes with unknown scene bounds. The code is available at https://github.com/laliwang/LCP-Fusion.

Keywords: Mapping, Localization, Robotics and Automation in Construction
Abstract: Ensuring an accurate and robust localization is one of the most significant problems in the field of mobile robotics. In this context, map-based localization methods utilizing 3D LiDARS for environmental perception are widely used. Even if there exist multiple promising techniques in this field, the majority of approaches can only guarantee an accurate and robust operation if there is no deviation between the map and the real surroundings. Consequently, state of the art localization methods frequently suffer from unreliable results or even complete failure in the case of changing environments. In this paper, we propose an efficient technique for a precise and robust maintenance of localization maps through realtime incorporation of 3D LiDAR scans. Our map update procedure is based on a novel way of estimating the interference between laserbeams and map contents, denoted as Ray-Bundle-Impact-Factor (RBIF). Our technique additionally solves the widespread problem of disruptive hole creation caused by discretization effects. Experiments on real-world as well as synthetic data demonstrate the precision and stability of our method under various challenging conditions and evaluate our approach in comparison to multiple SOTA map maintenance algorithms.

Keywords: Mapping, Range Sensing, Deep Learning for Visual Perception
Abstract: Recently, significant progress has been achieved in sensing real large-scale outdoor 3D environments, particularly by using modern acquisition equipment such as LiDAR sensors. Unfortunately, they are fundamentally limited in their ability to produce dense, complete 3D scenes. To address this issue, recent learning-based methods integrate neural implicit representations and optimizable feature grids to approximate surfaces of 3D scenes. However, naively fitting samples along raw LiDAR rays leads to noisy 3D mapping results due to the nature of sparse, conflicting LiDAR measurements. Instead, in this work we depart from fitting LiDAR data exactly, instead letting the network optimize a non-metric monotonic implicit field defined in 3D space. To fit our field, we design a learning system integrating a monotonicity loss that enables optimizing neural monotonic fields and leverages recent progress in large-scale 3D mapping. Our algorithm achieves high-quality dense 3D mapping performance as cap- tured by multiple quantitative and perceptual measures and visual results obtained for Mai City, Newer College, and KITTI benchmarks. The code of our approach is publicly available at https://github.com/artonson/deepmif.

Keywords: Mapping, SLAM
Abstract: Localization and mapping are critical tasks for various applications such as autonomous vehicles and robotics. The challenges posed by outdoor environments present particular complexities due to their unbounded characteristics. In this work, we present MM-Gaussian, a LiDAR-camera multi-modal fusion system for localization and mapping in unbounded scenes. Our approach is inspired by the recently developed 3D Gaussians, which demonstrate remarkable capabilities in achieving high rendering quality and fast rendering speed. Specifically, our system fully utilizes the geometric structure information provided by solid-state LiDAR to address the problem of inaccurate depth encountered when relying solely on visual solutions in unbounded, outdoor scenarios. Additionally, we utilize 3D Gaussian point clouds, with the assistance of pixel-level gradient descent, to fully exploit the color information in photos, thereby achieving realistic rendering effects.To further bolster the robustness of our system, we designed a relocalization module, which assists in returning to the correct trajectory in the event of a localization failure. Experiments conducted in multiple scenarios demonstrate the effectiveness of our method.

Keywords: Mapping, SLAM, Failure Detection and Recovery
Abstract: This paper addresses the failure detection and recovery problem in visual-inertial based Simultaneous Localization and Mapping (SLAM) systems for large-scale indoor environments. Camera and Inertial Measurement Unit (IMU) are popular choices for SLAM in many robotics tasks (e.g., navigation) due to their complementary sensing capabilities and low cost. However, vision has inherent challenges even in well-lit scenes, including motion blur, lack of features, or even accidental camera blockage. These failures can cause drifts to accumulate over time and can severely impact the scalability of existing solutions to large areas. To address these issues, we propose an automatic map generation service with (i) a failure detection method based on visual feature tracking quality using a health tracker which identifies and discards faulty measurements and (ii) a continuous session merging approach in SLAM. Taken together, this allows us to handle erroneous data without any manual intervention, and allows us to scale to extremely large spaces. The proposed system has been validated on benchmark datasets. Also, experimental results on multiple custom large-scale grocery stores, each between 1700 m^2 to 3700 m^2, and duration 60 to 80 minutes, are presented. Our approach shows the lowest error in all large-scale SLAM cases when compared with state-of-the-art visual-inertial SLAM packages, which often produce highly erroneous trajectories or lose track. Additionally, we provide dense 3D reconstruction with the presence of a depth camera by simply registering the point cloud from RGB-D image with respect to the SLAM generated trajectory -- and the quality of the reconstruction illustrates the efficacy of our proposed method.

Keywords: Autonomous Agents, View Planning for SLAM, Mapping
Abstract: The autonomous mapping of large-scale urban scenes presents significant challenges for autonomous robots. To mitigate the challenges, global planning, such as utilizing prior GPS trajectories from OpenStreetMap (OSM), is often used to guide the autonomous navigation of robots for mapping. However, due to factors like complex terrain, unexpected body movement, and sensor noise, the uncertainty of the robot's pose estimates inevitably increases over time, ultimately leading to the failure of robotic mapping. To address this issue, we propose a novel active loop closure procedure, enabling the robot to actively re-plan the previously planned GPS trajectory. The method can guide the robot to re-visit the previous places where the loop-closure detection can be performed to trigger the back-end optimization, effectively reducing errors and uncertainties in pose estimation. The proposed active loop closure mechanism is implemented and embedded into a real-time OSM-guided robot mapping framework. Empirical results on several large-scale outdoor scenarios demonstrate its effectiveness and promising performance.

Keywords: Data Sets for SLAM, Big Data in Robotics and Automation, Mapping
Abstract: Detecting moving events produced by moving objects is a crucial task in the realms of autonomous driving and mobile robots. Moving objects have the potential to create ghost artifacts in mapped environments and pose risks to autonomous navigation. LiDAR serves as a vital sensor for autonomous systems due to its ability to provide dense and precise range measurements. However, existing LiDAR datasets often lack sufficient discussion on the motion labeling of moving objects, containing only a limited representation of moving entities within a single scene. Furthermore, the methodologies for Moving Event Detection (MED) on LiDAR sensors have not been comprehensively explored or evaluated. To address these gaps, this study focuses on constructing a diverse LiDAR moving event dataset encompassing multiple scenes with a high density of moving objects. A thorough review of current MED techniques is conducted, followed by the establishment of a performance benchmark based on evaluating these methods using our dataset. Additionally, part sequences of the dataset are utilized to host an online MED competition, aimed at fostering collaboration within the research community and advancing related studies.

Keywords: Robot Audition, Mapping, Localization
Abstract: This paper addresses the challenge of acoustic scene mapping (ASM) in complex indoor environments with multiple sound sources. Unlike existing methods that rely on prior data association or SLAM frameworks, we propose a novel particle filter-based iterative framework, termed I-ASM, for ASM using a mobile robot equipped with a microphone array and LiDAR. I-ASM harnesses an innovative “implicit association” to align sound sources with Direction of Arrival (DoA) observations without requiring explicit pairing, thereby streamlining the mapping process. Given inputs including an occupancy map, DoA estimates from various robot positions, and corresponding robot pose data, I-ASM performs multi-source mapping through an iterative cycle of ``Filtering-Clustering-Implicit Associating''. The proposed framework has been tested in real-world scenarios with up to 10 concurrent sound sources, demonstrating its robustness against missing and false DoA estimates while achieving high-quality ASM results. To benefit the community, we open-source all the codes and data at https://github.com/AISLAB-sustech/Acoustic-Scene-Mapping

Keywords: RGB-D Perception, SLAM, Localization
Abstract: Previous attempts to integrate Neural Radiance Fields (NeRF) into the Simultaneous Localization and Mapping (SLAM) framework either rely on the assumption of static scenes or require the ground truth camera poses, which impedes their application in real-world scenarios. This paper proposes a time-varying representation to track and reconstruct the dynamic scenes. Firstly, two processes, a tracking process and a mapping process, are maintained simultaneously in our framework. In the tracking process, all input images are uniformly sampled and then progressively trained in a self-supervised paradigm. In the mapping process, we leverage motion masks to distinguish dynamic objects from the static background, and sample more pixels from dynamic areas. Secondly, the parameter optimization for both processes is comprised of two stages: the first stage associates time with 3D positions to convert the deformation field to the canonical field. The second stage associates time with the embeddings of the canonical field to obtain colors and a Signed Distance Function (SDF). Lastly, we propose a novel keyframe selection strategy based on the overlapping rate. Our approach is evaluated on two synthetic datasets and one real-world dataset, and the experiments validate that our method achieves competitive results in both tracking and mapping when compared to existing state-of-the-art NeRF-based dynamic SLAM systems.

Keywords: Semantic Scene Understanding, Mapping, Autonomous Vehicle Navigation
Abstract: Vectorized reconstruction and topological cognition of road structures are crucial for autonomous vehicles to handle complex scenes. Traditional frameworks rely heavily on high-definition (HD) maps, which place significant demands on storage, computation, and manual labor. To overcome these limitations, we introduce a lightweight Road Cognition and Automated Labeling (RCAL) system. It leverages lightweight road data captured from mass-produced vehicles to vectorize road elements and cognize their topology. RCAL compiles multi-trip data on cloud servers for enhanced accuracy and coverage, addressing the limitations of single-trip data. In the field of element extraction, we proposed a pivotal point priority sampling strategy that can balance the contradiction between road scale and processing efficiency. Additionally, traffic flow is utilized to enhance the accuracy of road topology cognition. With its impressive automation, reliability, and efficiency, RCAL stands as an advanced solution in the field. Our evaluations on the intersection dataset from the real world confirm that RCAL not only achieves comparable precision to traditional HD map labeling systems but also substantially reducing resource costs.

Keywords: Semantic Scene Understanding, Mapping, Deep Learning for Visual Perception
Abstract: Recently, progress in acquisition equipment such as LiDAR sensors has enabled sensing increasingly spacious outdoor 3D environments. Making sense of such 3D acquisitions requires fine-grained scene understanding, such as constructing instance-based 3D scene segmentations. Commonly, a neural network is trained for this task; however, this requires access to a large, densely annotated dataset, which is widely known to be challenging to obtain. To address this issue, in this work we propose to predict instance segmentations for 3D scenes in an unsupervised way, without relying on ground-truth annotations. To this end, we construct a learning framework consisting of two components: (1) a pseudo-annotation scheme for generating initial unsupervised pseudo-labels; and (2) a self-training algorithm for instance segmentation to fit robust, accurate instances from initial noisy proposals. To enable generating 3D instance mask proposals, we construct a weighted proxy-graph by connecting 3D points with edges integrating multi-modal image- and point-based self-supervised features, and perform graph- cuts to isolate individual pseudo-instances. We then build on a state-of-the-art point-based architecture and train a 3D instance segmentation model, resulting in significant refinement of initial proposals. To scale to arbitrary complexity 3D scenes, we design our algorithm to operate on local 3D point chunks and construct a merging step to generate scene-level instance segmentations. Experiments on the challenging SemanticKITTI benchmark demonstrate the potential of our approach, where it attains 13.3% higher Average Precision and 9.1% higher F1 score compared to the best-performing baseline. The code is publicly available at https://github.com/artonson/autoinst.

Keywords: Range Sensing, Sensor Fusion
Abstract: Inspired by the flash-to-bang theory, we develop FBDepth, the first passive audio-visual depth estimation framework. It is based on the difference between the time-of-flight (ToF) of the light and the sound. We formulate sound source depth estimation as an audio-visual event localization task for collision events. To approach decimeter-level depth accuracy, we design a coarse-to-fine pipeline to push the temporary localization accuracy from event-level to millisecond-level by aligning audio-visual correspondence and manipulating optical flow. FBDepth feeds the estimated visual timestamp together with the audio clip and objects visual features to regress the source depth. We use a mobile phone to collect 3.6K+ video clips with 24 different objects at up to 65m. FBDepth shows superior performance especially at a long range compared to monocular and stereo methods.

Keywords: Force and Tactile Sensing, Soft Sensors and Actuators, Soft Robot Applications
Abstract: Recent developments in vision-based tactile sensing offer a simple means to enable robots to perceive touch interactions. However, existing sensors are primarily designed for small-scale applications like robotic hands, lacking research on their integration for large-sized robot bodies that can be leveraged for safe human-robot interactions. This paper explores the utilization of the previously-developed vision-based tactile sensing link (called TacLink) with soft skin as a safety control mechanism, which can serve as an alternative to conventional rigid robot links and impact observers. We characterize the behavior of a robot integrated with the soft TacLink in response to collisions, particularly employing a reactive control strategy. The controller is primarily driven by tactile force information acquired from the soft TacLink sensor through a data-driven sim2real learning method. Compared with a standard rigid link, the results obtained from collision experiments also confirm the advantages of our "soft" solution in impact resilience and in facilitating controls that are difficult to achieve with a stiff robot body. This study can act as a benchmark for assessing the efficiency of soft tactile-sensitive skins in reactive collision responses and open new safety standards for soft skin-based collaborative robots in human-robot interaction scenarios.

Keywords: Failure Detection and Recovery, Robust/Adaptive Control, Reinforcement Learning
Abstract: Abstract— Reinforcement learning techniques are widely used when robots have to learn new tasks but they typically operate on action spaces defined by the joints of the robot. We present a contrasting approach where actions spaces are the trajectories of objects in the environment, requiring robots to discover events such as object changes and behaviors that must occur to accomplish the task. We show that this allows robots to learn faster, to learn semantic representations that can be communicated to humans, and to learn in a manner that does not depend on the robot itself, enabling low-cost policy transfer between different types of robots. Our demonstrations can be replicated using provided source code.

Keywords: Robot Safety, Collision Avoidance, Motion and Path Planning
Abstract: This paper proposes a LiDAR-based goal-seeking and exploration framework, addressing the efficiency of online obstacle avoidance in unstructured environments populated with static and moving obstacles. This framework addresses two significant challenges associated with traditional dynamic control barrier functions (D-CBFs): their online construction and the diminished real-time performance caused by utilizing multiple D-CBFs. To tackle the first challenge, the framework's perception component begins with clustering point clouds via the DBSCAN algorithm, followed by encapsulating these clusters with the minimum bounding ellipses (MBEs) algorithm to create elliptical representations. By comparing the current state of MBEs with those stored from previous moments, the differentiation between static and dynamic obstacles is realized, and the Kalman filter is utilized to predict the movements of the latter. Such analysis facilitates the D-CBF's online construction for each MBE. To tackle the second challenge, we introduce buffer zones, generating Type-II D-CBFs online for each identified obstacle. Utilizing these buffer zones as activation areas substantially reduces the number of D-CBFs that need to be activated. Upon entering these buffer zones, the system prioritizes safety, autonomously navigating safe paths, and hence referred to as the exploration mode. Exiting these buffer zones triggers the system's transition to goal-seeking mode. We demonstrate that the system's states under this framework achieve safety and asymptotic stabilization. Experimental results in simulated and real-world environments have validated our framework's capability, allowing a LiDAR-equipped mobile robot to efficiently and safely reach the desired location within dynamic environments containing multiple obstacles.

Keywords: Robot Safety, Reinforcement Learning, Robust/Adaptive Control

Keywords: Robot Safety, Hybrid Logical/Dynamical Planning and Verification, Autonomous Vehicle Navigation
Abstract: Autonomous Vehicles (AVs) are often tested in simulation to estimate the probability they violate safety specifications. Two common issues arise when using existing techniques to produce this estimation: If violations are rare, Monte-Carlo sampling can fail to produce efficient estimates; if simulation horizons are too long, importance samplers (which learn distributions from past simulations) can fail to converge. This paper addresses both issues by interleaving rare-event sampling techniques with online specification monitoring algorithms. We use importance splitting to decompose simulations into partial trajectories, then calculate the distance of those partial trajectories to failure by leveraging robustness metrics from Signal Temporal Logic (STL). By caching those partial robustness metric values, we can efficiently re-use computations across multiple sampling stages. Our experiments on an interstate lane-change scenario show our method is viable for testing simulated AV-pipelines, efficiently estimating failure probabilities for STL specifications based on real traffic rules. We produce better estimates than Monte-Carlo and importance sampling in fewer simulations.

Keywords: Robot Safety, Humanoid and Bipedal Locomotion
Abstract: We study the problem of natural language-based control of dynamic bipedal locomotion from the perspective of operational robustness and hardware safety. Existing work on natural language-based robot control has focused on episodic command execution for stable robot platforms, such as fixed-based manipulators in table-top scenarios. These scenarios feature non-overlapping phases of instruction and execution, with execution mishaps usually posing no threat to the robot safety. This allows for non-trivial failure rates to be acceptable. In contrast, our work involves indistinguishable instruction and execution stages for a dynamically unstable robot where execution failures can harm the robot. For example, interrupting a bipedal robot with a new instruction in certain states may cause it to fall. Our first contribution is to design and train a natural language based controller for the bipedal robot Cassie that can take in new language commands at any time. Our second contribution is to introduce a protocol for evaluating the robustness to interruptions of such controllers and evaluating the learned controller in simulation under different interruption distributions. Our third contribution is to learn a detector for interruptions that are likely to lead to failure and to integrate that detector into a failure mitigation strategy. Overall, our results show that interruptions can lead to non-trivial failure rates for the original controller and that the proposed mitigation strategy can help to significantly reduce that rate.

Keywords: Robot Safety, Deep Learning Methods, Reinforcement Learning
Abstract: We introduce the Safe Offline-to-Online Multi-Agent Decision Transformer (SO2-MADT), an innovative framework that revolutionizes safety considerations in Multi-agent Reinforcement Learning (MARL) through a novel sequence modeling approach. Leveraging the dynamic capabilities inherent in Decision Transformers, our methodology seamlessly incorporates safety protocols as a cornerstone element, ensuring secure operations throughout both the offline pre-training phase and the adaptive online fine-tuning phase. At the core of our framework lie two pivotal innovations: the Safety-To-Go (STG) token, embedding safety at a macro level, and the Agent Prioritization Module (APM), facilitating explicit credit assignment at a micro level. Through extensive testing against the challenging environments of the StarCraft Multi-Agent Challenge (SMAC) and Multi-agent MuJoCo, our SO2-MADT not only excels in offline pre-training but also demonstrates superior performance during online fine-tuning, without any degradation in performance. The implications of our work provide a pathway for deployment in critical real-world applications where safety is paramount and non-negotiable. The code is available at https://github.com/shahaamirbader/SO2-MADT.

Keywords: Robot Safety, Underactuated Robots, Optimization and Optimal Control
Abstract: This paper presents a control barrier function (CBF) for systems described by differential-algebraic equations, and applies the method to guarantee the safety of a two-link flexible-link manipulator. The two main contributions of the paper are: a) an extension of CBFs to systems governed by differential-algebraic equations; b) a framework simulation of flexible-link robots in a floating frame of reference formulation (FFRF) finite element method (FEM). Numerical simulations demonstrate the minimally invasive safety control of a flexible two-link manipulator with position constraints through CBF quadratic programming without converting the differential-algebraic equations to a control-affine system.

Keywords: Virtual Reality and Interfaces, Robot Safety, Aerial Systems: Perception and Autonomy
Abstract: We present a high-fidelity Mixed Reality sensor emulation framework for testing and evaluating the resilience of Unmanned Aerial Vehicles (UAVs) against false data injection (FDI) attacks. The proposed approach can be utilized to assess the impact of FDI attacks, benchmark attack detector performance, and validate the effectiveness of mitigation/reconfiguration strategies in single-UAV and UAV swarm operations. Our Mixed Reality framework leverages high-fidelity simulations of Gazebo and a Motion Capture system to emulate proprioceptive (e.g., GNSS) and exteroceptive (e.g., camera) sensor measurements in real-time. We propose an empirical approach to faithfully recreate signal characteristics such as latency and noise in these measurements. Finally, we illustrate the efficacy of our proposed framework through a Mixed Reality experiment consisting of an emulated GNSS attack on an actual UAV, which (i)demonstrates the impact of false data injection attacks on GNSS measurements and (ii) validates a mitigation strategy utilizing a distributed camera network developed in our previous work. Our open-source implementation is available at https://github.com/CogniPilot/mixed_sense

Keywords: Bimanual Manipulation, Reinforcement Learning, Robot Safety
Abstract: Bimanual dexterous manipulation in robotics, essential for a wide range of applications, addresses the critical challenge of balancing intricate operational capabilities with assured safety and reliability. While Safe Reinforcement Learning is integral to the robustness of robotic systems, the area of safe multi-agent reinforcement learning (MARL), cooperative control of multiple robots has been scarcely studied. In this study, we explore MARL for safe cooperative control with multiple robot hands. Each robot must follow individual and collective safety guidelines to ensure safe team actions. However, the non-stationarity inherent in current algorithms hinders the precise updating of strategies to satisfy these safety constraints effectively. In this paper, we propose Multi-Agent Constrained Proximal Advantage Optimization (MACPAO), which considers the sequence of agent updates and integrates non-stationarity into sequential update schemes. This algorithm ensures consistent improvement in both rewards and adherence to safety constraints in each iteration. We tested MACPAO on various tasks with safety constraints and demonstrated that it outperforms other MARL algorithms in balancing reward enhancement and safety compliance. Supplementary materials and code are available at the provided link https://github.com/YONEX4090/MultiSafeHand.git.

Keywords: Control Architectures and Programming, Formal Methods in Robotics and Automation, Optimization and Optimal Control
Abstract: This paper introduces CBFkit, a Python/ROS toolbox for safe robotics planning and control under uncertainty. The toolbox provides a general framework for designing control barrier functions for mobility systems within both deterministic and stochastic environments. It can be connected to the ROS open-source robotics middleware, allowing for the setup of multi-robot applications, encoding of environments and maps, and integrations with predictive motion planning algorithms. Additionally, it offers multiple CBF variations and algorithms for robot control. The CBFKit is demonstrated on the Toyota Human Support Robot (HSR) in both simulation and in physical experiments.

Keywords: Failure Detection and Recovery, AI-Enabled Robotics, AI-Based Methods
Abstract: Recognizing failures during task execution and implementing recovery procedures is challenging in robotics. Traditional approaches rely on the availability of extensive data or a tight set of constraints, while more recent approaches leverage large language models (LLMs) to verify task steps and replan accordingly. However, these methods often operate offline, necessitating scene resets and incurring in high costs. This paper introduces RECOVER, a neuro-symbolic framework for online failure identification and recovery. By integrating ontologies, logical rules, and LLM-based planners, RECOVER exploits symbolic information to enhance the ability of LLMs to generate recovery plans and also to decrease the associated costs. In order to demonstrate the capabilities of our method in a simulated kitchen environment, we introduce ONTOTHOR, an ontology describing the AI2Thor simulator setting. Empirical evaluation shows that ONTOTHOR’s logical rules accurately detect all failures in the analyzed tasks, and that RECOVER considerably outperforms, for both failure detection and recovery, a baseline method reliant solely on LLMs. Supplementary material, including the ONTOTHOR ontology, is available at: https://recover-ontothor.github.io.

Keywords: Soft Robot Applications, Robot Safety
Abstract: Soft growing vine robots show great potential for navigation and decontamination tasks in the nuclear industry. This paper introduces a novel hybrid continuum-eversion robot designed to address certain challenges in relation to navigating and operating within pipe networks and enclosed remote vessels. The hybrid robot combines the flexibility of a soft eversion robot with the precision of a continuum robot at its tip, allowing for controlled steering and movement in hard to access and/or complex environments. The design enables the delivery of sensors, liquids, and aerosols to remote areas, supporting remote decontamination activities.

This paper outlines the design and construction of the robot and the methods by which it achieves selective steering. We also include a comprehensive review of current related work in eversion robotics, as well as other steering devices and actuators currently under research, which underpin this novel active steering approach. This is followed by an experimental evaluation that demonstrates the robot’s real-world capabilities in delivering liquids and aerosols to remote locations. The experiments reveal successful outcomes, with over 95% success in precision spraying tests. The paper concludes by discussing future work alongside limitations in the current design, ultimately showcasing its potential as a solution for remote decontamination operations in the nuclear industry.

Keywords: Engineering for Robotic Systems, AI-Based Methods
Abstract: The ubiquitous deployment of robots across diverse domains, from industrial automation to personal care, underscores their critical role in modern society. However, this growing dependence has also revealed security vulnerabilities. An attack vector involves the deployment of malicious software (malware) on robots, which can cause harm to robots themselves, users, and even the surrounding environment. Machine learning approaches, particularly supervised ones, have shown promise in malware detection by building intricate models to identify known malicious code patterns. However, these methods are inherently limited in detecting unseen or zero-day malware variants as they require regularly updated massive datasets that might be unavailable to robots. To address this challenge, we introduce RoboGuardZ, a novel malware detection framework based on zero-shot learning for robots. This approach allows RoboGuardZ to identify unseen malware by establishing relationships between known malicious code and benign behaviors, allowing detection even before the code executes on the robot. To ensure practical deployment in resource-constrained robotic hardware, we employ a unique parallel structured pruning and quantization strategy that compresses the RoboGuardZ detection model by 37.4% while maintaining its accuracy. This strategy reduces the size of the model and computational demands, making it suitable for real-world robotic systems. We evaluated RoboGuardZ on a recent dataset containing real-world binary executables from multi-sensor autonomous car controllers. The framework was deployed on two popular robot embedded hardware platforms. Our results demonstrate an average detection accuracy of 94.25% and a low false negative rate of 5.8% with a minimal latency of 20 ms, which demonstrates its effectiveness and practicality.

Keywords: Embedded Systems for Robotic and Automation, AI-Based Methods
Abstract: Zero-day vulnerabilities pose a significant challenge to robot cyber-physical systems (CPS). Attackers can exploit software vulnerabilities in widely-used robotics software, such as the Robot Operating System (ROS), to manipulate robot behavior, compromising both safety and operational effectiveness. The hidden nature of these vulnerabilities requires strong defense mechanisms to guarantee the safety and dependability of robotic systems. In this paper, we introduce RoboCop, a cyber-physical attack detection framework designed to protect robots from zero-day threats. RoboCop leverages static software features in the pre-execution analysis along with runtime state monitoring to identify attack patterns and deviations that signal attacks, thus ensuring the robot's operational integrity. We evaluated RoboCop on the F1-tenth autonomous car platform. It achieves a 93% detection accuracy against a variety of zero-day attacks targeting sensors, actuators, and controller logic. Importantly, in on-robot deployments, it identifies attacks in less than 7 seconds with a 12% computational overhead.

Keywords: Computational Geometry, Contact Modeling, Simulation and Animation
Abstract: Collision detection is a fundamental problem across various fields such as robotics, physical simulation, and computer graphics. While numerous studies have provided efficient solutions, based on the well-known Gilbert, Johnson, and Keerthi (GJK) algorithm and Expanding Polytope Algorithm (EPA), existing methods utilizing GJK-EPA often struggle with smooth strictly convex shapes like ellipsoids. This paper proposes a novel approach to the collision detection problem converting it to a problem compatible with an unconstrained Riemannian optimization problem. Moreover, we presents a specific method of solving the problem based on twice differentiable support functions and the Riemannian trust region (RTR) method. The method exhibits fast and robust convergence rate, leveraging the well-established theory of Riemannian optimization. In addition, it offers the capability to compute derivatives of resultant contact features. The evaluation studies comparing our method to GJK-EPA method are done with pre-defined primitive shapes. Additionally, a test result with several more complex shapes is demonstrated exhibiting the method's effectiveness and applicability.

Keywords: Force and Tactile Sensing, Perception for Grasping and Manipulation
Abstract: Cross-Modal Retrieval (CMR), which retrieves relevant items from one modality (e.g., audio) given a query in another modality (e.g., visual), has undergone significant advancements in recent years. This capability is crucial for robots to integrate and interpret information across diverse sensory inputs. However, the retrieval space in existing robotic CMR approaches often consists of only one modality, which limits the robot’s performance. In this paper, we propose a novel CMR model that incorporates three different modalities, i.e., visual, audio and tactile, for enhanced multi-modal object retrieval, named as VAT-CMR. In this model, multi-modal representations are first fused to provide a holistic view of object features. To mitigate the semantic gaps between representations of different modalities, a dominant modality is then selected during the classification training phase to improve the distinctiveness of the representations, so as to improve the retrieval performance. To evaluate our proposed approach, we conducted a case study and the results demonstrate that our VAT-CMR model surpasses competing approaches. Further, our proposed dominant modality selection significantly enhances cross-retrieval accuracy.

Keywords: Sensor Fusion, Computer Vision for Transportation
Abstract: Panoptic Driving Perception (PDP) is critical for the autonomous navigation of Unmanned Surface Vehicles (USVs). A PDP model typically integrates multiple tasks, necessitating the simultaneous and robust execution of various perception tasks to facilitate downstream path planning. The fusion of visual and radar sensors is currently acknowledged as a robust and cost-effective approach. However, most existing research has primarily focused on fusing visual and radar features dedicated to object detection or utilizing a shared feature space for multiple tasks, neglecting the individual representation differences between various tasks. To address this gap, we propose a pair of Asymmetric Fair Fusion (AFF) modules with favorable explainability designed to efficiently interact with independent features from both visual and radar modalities, tailored to the specific requirements of object detection and semantic segmentation tasks. The AFF modules treat image and radar maps as irregular point sets and transform these features into a crossed-shared feature space for multitasking, ensuring equitable treatment of vision and radar point cloud features. Leveraging AFF modules, we propose a novel and efficient PDP model, ASY-VRNet, which processes image and radar features based on irregular super-pixel point sets. Additionally, we propose an effective multi-task learning method specifically designed for PDP models. Compared to other lightweight models, ASY-VRNet achieves state-of-the-art performance in object detection, semantic segmentation, and drivable-area segmentation on the WaterScenes benchmark. Our project is publicly available at https://github.com/GuanRunwei/ASY-VRNet.

Keywords: Sensor Fusion, SLAM, Mapping
Abstract: This paper presents a Kalman filter based LiDAR-Inertial-Leg Odometry (KLILO) system for legged robots to navigate in challenging environments. In particular, we employ the iterated error-state extended Kalman filter framework on manifolds to fuse measurements from the inertial measurement unit (IMU), LiDAR, joint encoders, and contact force sensors in a tightly coupled manner. To assess the performance of KLILO, we build a dataset that encompasses intricate environments with challenging conditions such as dynamic objects and deformable terrains. The results demonstrate that our algorithm can provide efficient and reliable localization in all tests. It exhibits an average improvement of around 40% in positioning accuracy compared to the baselines. Furthermore, we validate KLILO in a challenging navigation task on a real robot, where the LiDAR encounters ineffective measurements.

Keywords: Sensor Fusion, Marine Robotics, Autonomous Vehicle Navigation
Abstract: Determining the position of obstacles is crucial for unmanned vehicles, and, to achieve this, cameras and radar sensors are widely utilized. However, establishing correlation between two or more sensors proves challenging in the dynamically changing maritime environment. To solve these issues, we propose the AnytimeFusion algorithm. The key innovation of AnytimeFusion lies in the utilization of a parameter-free method that does not require accurate sensor alignment and calibration. The algorithm consists of the following four stages. First, calibration targets are selected in the maritime environment based on segmentation images. Second, radar and camera data are pre-fused to model the correlation of azimuth information. After completing the auto-calibration stages, Inverse Perspective Mapping (IPM) is employed to integrate the coordinate systems of the two sensors. To determine the parameters for this integration, optimization based on the Particle Swarm Optimization (PSO) method is employed. Finally, an Error Polygon for the positions of the camera and radar is generated, and sensor fusion is carried out based on this information. We validated our method through experiments conducted on real ships in complex maritime environments, achieving an average accuracy of 95.7%.

Keywords: Sensor Fusion, Recognition, Computer Vision for Automation
Abstract: Optical sensors, such as the Far Infrared (FIR) sensor, have demonstrated advantages over traditional imaging. For example, 3D reconstruction in the FIR field captures the heat distribution of a scene that is invisible to RGB, aiding various applications like gas leak detection. However, less texture information and challenges in acquiring FIR frames hinder the reconstruction process. Given that implicit neural representations (INRs) can integrate geometric information across different sensors, we propose Implicit Neural Fusion (INF) of RGB and FIR for 3D reconstruction of invisible scenes in the FIR field. Our method first obtains a neural density field of objects from RGB frames. Then, with the trained object density field, a separate neural density field of gases is optimized using limited view inputs of FIR frames. Our method not only demonstrates outstanding reconstruction quality in the FIR field through extensive experiments but also can isolate the geometric information of the invisible, offering a new dimension of scene understanding.

Keywords: Sensor Fusion, Intelligent Transportation Systems, Autonomous Vehicle Navigation
Abstract: We present a multimodal traffic light state detection using vision and sound, from the viewpoint of a quadruped robot navigating in urban settings. This is a challenging problem because of the visual occlusions and noise from robot locomotion. Our method combines features from raw audio with the ratios of red and green pixels within bounding boxes, identified by established vision-based detectors. The fusion method aggregates features across multiple frames in a given timeframe, increasing robustness and adaptability. Results show that our approach effectively addresses the challenge of visual occlusion and surpasses the performance of single-modality solutions when the robot is in motion. This study serves as a proof of concept, highlighting the significant, yet often overlooked, potential of multi-modal perception in robotics.

Keywords: Sensor Fusion, Range Sensing, Sensor Networks
Abstract: We present a tracking system for relative positioning that can operate on entirely moving tracking nodes without the need for stationary anchors. Each node embeds a 9-DOF magnetic and inertial measurement unit and a single-antenna ultra-wideband radio. We introduce a multi-stage filtering pipeline through which our system estimates the relative layout of all tracking nodes within the group. The key novelty of our method is the integration of a custom Extended Kalman filter (EKF) with a refinement step via multidimensional scaling (MDS). Our method integrates the MDS output back into the EKF, thereby creating a dynamic feedback loop for more robust estimates. We complement our method with UWB ranging protocol that we designed to allow tracking nodes to opportunistically join and leave the group. In our evaluation with constantly moving nodes, our system estimated relative positions with an error of 10.2cm (in 2D) and 21.7cm (in 3D), including obstacles that occluded the line of sight between tracking nodes. Our approach requires no external infrastructure, making it particularly suitable for operation in environments where stationary setups are impractical.

Keywords: Sensor Fusion, Visual Learning, Semantic Scene Understanding
Abstract: In this work, we focus on unsupervised vision-language-action mapping in the area of robotic manipulation. Recently, multiple approaches employing pre-trained large language and vision models have been proposed for this task. However, they are computationally demanding and require careful fine-tuning of the produced output. A more lightweight alternative would be the implementation of multimodal Variational Autoencoders (VAEs) which can extract the latent features of the data and integrate them into a joint representation, as has been demonstrated mostly on image-image or image-text data for the state-of-the-art models. Here, we explore whether and how multimodal VAEs can be employed in unsupervised robotic manipulation tasks in a simulated environment. Based on the results obtained, we propose a model-invariant training alternative that improves the models' performance in a simulator by up to 55%. Moreover, we systematically evaluate the challenges raised by individual tasks, such as object or robot position variability, number of distractors, or task length. Our work thus also sheds light on the potential benefits and limitations of using the current multimodal VAEs for unsupervised learning of robotic motion trajectories based on vision and language.

Keywords: Sensor Fusion, SLAM, Deep Learning for Visual Perception
Abstract: Multimodal sensor fusion has been successfully utilized in many odometry and localization methods as it increases both estimate accuracy and robustness in application scenarios. To address the challenge of odometry under varying-weather conditions, we propose a novel visual 4D radar fusion based odometry in an unsupervised deep learning approach. In our method, we adopt transformer-based cascaded decoders to facilitate efficient feature extraction of images and radar point clouds. Considering that radars are weather-agnostic and information-rich cameras are susceptible to adverse weathers, we deliberately introduce an adaptive feature fusion strategy via the attention mechanism, in which the attention shifts dynamically to adapt to changing weather conditions based on the amount of information content in image features. Through extensive comparative experiments, our method surpasses different state-of-the-art single-modal odometry estimation methods. Our code and trained model will be released publicly.

Keywords: Sensor Fusion, Mapping, AI-Enabled Robotics
Abstract: Autonomous mobile robots are an increasingly integral part of modern factory and warehouse operations. Obstacle detection, avoidance and path planning are critical safety-relevant tasks, which are often solved using expensive LiDAR sensors and depth cameras. We propose to use costeffective low-resolution ranging sensors, such as ultrasonic and infrared time-of-flight sensors by developing VIRUS-NeRF - Vision, InfraRed, and UltraSonic based Neural Radiance Fields. Building upon Instant Neural Graphics Primitives with a Multiresolution Hash Encoding (Instant-NGP), VIRUS-NeRF incorporates depth measurements from ultrasonic and infrared sensors and utilizes them to update the occupancy grid used for ray marching. Experimental evaluation in 2D demonstrates that VIRUS-NeRF achieves comparable mapping performance to LiDAR point clouds regarding coverage. Notably, in small environments, its accuracy aligns with that of LiDAR measurements, while in larger ones, it is bounded by the utilized ultrasonic sensors. An in-depth ablation study reveals that adding ultrasonic and infrared sensors is highly effective when dealing with sparse data and low view variation. Further, the proposed occupancy grid of VIRUS-NeRF improves the mapping capabilities and increases the training speed by 46% compared to Instant-NGP. Overall, VIRUS-NeRF presents a promising approach for cost-effective local mapping in mobile robotics, with potential applications in safety and navigation tasks. The code can be found at https://github.com/ethz-asl/virus_nerf.

Keywords: Sensor Fusion, Visual-Inertial SLAM, Visual Tracking
Abstract: In this paper, we propose a monocular event-inertial odometry that incorporates an adaptive decay kernel-based time surface with a polarity-aware tracking method. Event cameras have garnered considerable attention because of their advantages over traditional cameras in low power consumption, high dynamic range, and no motion blur. To extract texture information from asynchronous events, our odometry implements the Time Surface based on adaptive decay, and the adaptive decay can adapt to the dynamic characteristics of the event stream and enhance the representation of environmental textures. Moreover, polarity-weighted time surfaces suffer from event polarity shifts in the motion direction change. To mitigate its adverse effects on feature tracking, we optimize the tracking process by incorporating an additional polarity-inverted time surface to enhance the robustness. We compare our method with visual-intertial and event-inertial odometry methods in terms of accuracy. Additionally, we conduct ablation experiments on both decays, parameter settings and the polarity-aware tracking method. The experimental results demonstrate better performance over the state-of-the-art methods, along with competitive outcomes across various datasets.

Keywords: Sensor Fusion, Computer Vision for Automation, Object Detection, Segmentation and Categorization
Abstract: Multimodal image pairs can make object detection more reliable in challenging environments, so RGB-T object detection has gained extensive attention over the past decade. To alleviate the complementarity of the visible and thermal modality, we propose a novel lightweight Feature Enhancement-fusion Module (FEM), which is composed of the Channel Enhancement-fusion Unit (CEU) and Spatial Enhancement-fusion Unit (SEU) by extending the attention mechanism to operate on two modalities. CEU is used to exploit the complementarity and alleviate the data imbalance by combining internal and global channel attention. Additionally, SEU is utilized to guide the model to pay more attention to the regions of interest. By incorporating FEM, enhanced and fused features are obtained, leading to improved performance. The effectiveness and generalizability of FEM are validated by two public datasets and our proposed DCSANet achieves competitive performance while maintaining high speed (+%7.0 on LLVIP and +1.2% on FLIR in mAP). Moreover, we conducted ablation experiments to verify the effectiveness of the proposed operators.