Keywords: Optimization and Optimal Control, Embedded Systems for Robotic and Automation
Abstract: Model-predictive control (MPC) is a powerful tool for controlling highly dynamic robotic systems subject to complex constraints. However, MPC is computationally demanding, and is often impractical to implement on small, resource-constrained robotic platforms. We present TinyMPC, a high-speed MPC solver with a low memory footprint targeting the microcontrollers common on small robots. Our approach is based on the alternating direction method of multipliers (ADMM) and leverages the structure of the MPC problem for efficiency. We demonstrate TinyMPC’s effectiveness by benchmarking against the state-of-the-art solver OSQP, achieving nearly an order of magnitude speed increase, as well as through hardware experiments on a 27 gram quadrotor, demonstrating high-speed trajectory tracking and dynamic obstacle avoidance.

Keywords: Biological Cell Manipulation, Mobile Manipulation, Automation at Micro-Nano Scales
Abstract: This study proposes a novel movable microfluidic chip in which a microfluidic chip is integrated into a robotic manipulator for manipulating oocytes. The microfluidic device has the ability to release a single cell with gap effect. The robotic manipulator can control the position of the microfluidic chip. The microfluidic chip with a pipette tip is directly fabricated using 3D printing. Xenopus oocyte was used in the experiment. When oocytes move from bottom side of the channel to the tip side, they generate gaps between each other. The gap distance can reach about 16 times the diameter of oocyte. In addition, a capacitive sensor was used to detect oocytes in the manipulation processes. The results showed that oocytes were successfully released one by one with no deformation in shape using the movable microfluidic chip. The method has significant advantages in biomedicine engineering and micro-nano-manipulation.

Keywords: Robotics in Hazardous Fields, Industrial Robots, Computer Vision for Automation
Abstract: We propose an interpretable framework for reading analog gauges that is deployable on real world robotic systems. Our framework splits the reading task into distinct steps, such that we can detect potential failures at each step. Our system needs no prior knowledge of the type of gauge or the range of the scale and is able to extract the units used. We show that our gauge reading algorithm is able to extract readings with a relative reading error of less than 2%.

Keywords: Robust/Adaptive Control
Abstract: Parameter convergence in adaptive control is crucial for improving the stability and robustness of robotic systems. Nevertheless, a stringent condition named persistent excitation (PE) needs to be satisfied to ensure parameter convergence in the conventional adaptive robot control. Composite learning robot control (CLRC) is an innovative methodology that guarantees parameter convergence under a condition of interval excitation (IE) that is strictly weaker than PE. This paper puts forward a time-division multi-channel (TDMC) CLRC strategy such that parameter convergence is achieved even without the IE condition. In the TDMC mechanism, a filtered regressor is integrated with multiple time intervals to generate a generalized prediction error for parameter update, such that excitation information of regressor channels at different instants is exploited more effectively and efficiently to achieve fast and accurate parameter estimation. Global exponential stability with parameter convergence of the closed-loop system is achieved under a partial IE condition that is much weaker than IE. Experiments on a collaborative robot with 7 degrees of freedom have demonstrated the superiority of the proposed approach in both parameter estimation and trajectory tracking compared to start-of-the-art approaches.

Keywords: Dynamics, Hybrid Logical/Dynamical Planning and Verification, Motion Control
Abstract: Estimating the region of attraction RoA for a robot controller is essential for safe application and controller composition. Many existing methods require a closed-form expression that limit applicability to data-driven controllers. Methods that operate only over trajectory rollouts tend to be data-hungry. In prior work, we have demonstrated that topological tools based on Morse Graphs (directed acyclic graphs that combinatorially represent the underlying nonlinear dynamics) offer data-efficient RoA estimation without needing an analytical model. They struggle, however, with high-dimensional systems as they operate over a state-space discretization. This paper presents Morse Graph-aided discovery of Regions of Attraction in a learned Latent Space (MORALS). The approach combines auto-encoding neural networks with Morse Graphs. MORALS shows promising predictive capabilities in estimating attractors and their RoAs for data-driven controllers operating over high-dimensional systems, including a 67-dim humanoid robot and a 96-dim 3-fingered manipulator. It first projects the dynamics of the controlled system into a learned latent space. Then, it constructs a reduced form of Morse Graphs representing the bistability of the underlying dynamics, i.e., detecting when the controller results in a desired versus an undesired behavior. The evaluation on high-dimensional robotic datasets indicates data efficiency in RoA estimation.

Keywords: Legged Robots, Sensorimotor Learning, Task and Motion Planning
Abstract: Autonomous robots must navigate reliably in unknown environments even under compromised exteroceptive perception, or perception failures. Such failures often occur when harsh environments lead to degraded sensing, or when the perception algorithm misinterprets the scene due to limited generalization. In this paper, we model perception failures as invisible obstacles and pits, and train a reinforcement learning (RL) based local navigation policy to guide our legged robot. Unlike previous works relying on heuristics and anomaly detection to update navigational information, we train our navigation policy to reconstruct the environment information in the latent space from corrupted perception and react to perception failures end-to-end. To this end, we incorporate both proprioception and exteroception into our policy inputs, thereby enabling the policy to sense collisions on different body parts and pits, prompting corresponding reactions. We validate our approach in simulation and on the real quadruped robot ANYmal running in real-time (<10 ms CPU inference). In a quantitative comparison with existing heuristic-based locally reactive planners, our policy increases the success rate over 30% when facing perception failures. Project Page: https://bit.ly/45NBTuh.

Keywords: Vision-Based Navigation, AI-Enabled Robotics, Semantic Scene Understanding
Abstract: Understanding how humans leverage semantic knowledge to navigate unfamiliar environments and decide where to explore next is pivotal for developing robots capable of human-like search behaviors. We introduce a zero-shot navigation approach, Vision-Language Frontier Maps (VLFM), which is inspired by human reasoning and designed to navigate towards unseen semantic objects in novel environments. VLFM builds occupancy maps from depth observations to identify frontiers, and leverages RGB observations and a pre-trained vision-language model to generate a language-grounded value map. VLFM then uses this map to identify the most promising frontier to explore for finding an instance of a given target object category. We evaluate VLFM in photo-realistic environments from the Gibson, Habitat-Matterport 3D (HM3D), and Matterport 3D (MP3D) datasets within the Habitat simulator. Remarkably, VLFM achieves state-of-the-art results on all three datasets as measured by success weighted by path length (SPL) for the Object Goal Navigation task. Furthermore, we show that VLFM's zero-shot nature enables it to be readily deployed on real-world robots such as the Boston Dynamics Spot mobile manipulation platform. We deploy VLFM on Spot and demonstrate its capability to efficiently navigate to target objects within an office building in the real world, without any prior knowledge of the environment. The accomplishments of VLFM underscore the promising potential of vision-language models in advancing the field of semantic navigation. Videos of real world deployment can be viewed at naoki.io/vlfm.

Keywords: Machine Learning for Robot Control, Reinforcement Learning, Deep Learning Methods
Abstract: Geometric regularity, which leverages data symmetry, has been successfully incorporated into deep learning architectures such as CNNs, RNNs, GNNs, and Transformers. While this concept has been widely applied in robotics to address the curse of dimensionality when learning from high-dimensional data, the inherent reflectional and rotational symmetry of robot structures has not been adequately explored. Drawing inspiration from cooperative multi-agent reinforcement learning, we introduce novel network structures for single-agent control learning that explicitly capture these symmetries. Moreover, we investigate the relationship between the geometric prior and the concept of Parameter Sharing in multi-agent reinforcement learning. Last but not the least, we implement the proposed framework in online and offline learning methods to demonstrate its ease of use. Through experiments conducted on various challenging continuous control tasks on simulators and real robots, we highlight the significant potential of the proposed geometric regularity in enhancing robot learning capabilities.

Keywords: Humanoid and Bipedal Locomotion, Legged Robots, Reinforcement Learning
Abstract: Reinforcement learning (RL) for bipedal locomotion has recently demonstrated robust gaits over moderate terrains using only proprioceptive sensing. However, such blind controllers will fail in environments where robots must anticipate and adapt to local terrain, which requires visual perception. In this paper, we propose a fully-learned system that allows bipedal robots to react to local terrain while maintaining commanded travel speed and direction. Our approach first trains a controller in simulation using a heightmap expressed in the robot's local frame. Next, data is collected in simulation to train a heightmap predictor, whose input is the history of depth images and robot states. We demonstrate that with appropriate domain randomization, this approach allows for successful sim-to-real transfer with no explicit pose estimation and no fine-tuning using real-world data. To the best of our knowledge, this is the first example of sim-to-real learning for vision-based bipedal locomotion over challenging terrains.

Keywords: Deep Learning Methods, Vision-Based Navigation
Abstract: Robotic learning for navigation in unfamiliar environments needs to provide policies for both task-oriented navigation (i.e., reaching a goal that the robot has located), and task-agnostic exploration (i.e., searching for a goal in a novel setting). Typically, these roles are handled by separate models, for example by using subgoal proposals, planning, or separate navigation strategies. In this paper, we describe how we can train a single unified diffusion policy to handle both goal-directed navigation and goal-agnostic exploration, with the latter providing the ability to search novel environments, and the former providing the ability to reach a user-specified goal once it has been located. We show that this unified policy results in better overall performance when navigating to visually indicated goals in novel environments, as compared to approaches that use subgoal proposals from generative models, or prior methods based on latent variable models. We instantiate our method by using a large-scale Transformer- based policy trained on data from multiple ground robots, with a diffusion model decoder to flexibly handle both goal- conditioned and goal-agnostic navigation. Our experiments, conducted on a real-world mobile robot platform, show effective navigation in unseen environments in comparison with five alternative methods, and demonstrate significant improvements in performance and lower collision rates, despite utilizing smaller models than state-of-the-art approaches.

Keywords: Planning under Uncertainty, Integrated Planning and Learning, Multi-Robot SLAM
Abstract: Multi-agent exploration of a bounded 3D environment with unknown initial positions of agents is a challenging problem. It requires quickly exploring the environments and robustly merging the sub-maps built by the agents. Most of the current exploration strategies directly merge two sub-maps built by different agents when a single frame of overlap is detected, which can lead to incorrect merging due to the false-positive detection of the overlap and is thus not robust. Meanwhile, some state-of-the-art place recognition methods use sequence

matching for more robust data association. However, naively applying these sequence-based matching methods to multi-agent exploration may require one agent to repeat a large amount of another agent's history trajectory so that a sequence of matched observation can be established, which reduces the overall exploration time efficiency. To intelligently balance the robustness of sub-map merging and exploration

efficiency, we develop a new approach for lidar-based multi-agent exploration, which can direct one agent to repeat another agent's trajectory adaptively based on the quality indicator of the sub-map merging process. Additionally, our approach extends the recent single-agent hierarchical exploration strategy to multiple agents cooperatively by planning for agents with merged sub-maps to further improve exploration efficiency. Our experiments show that our approach is up to 50% more efficient than the baselines on average while merging

Keywords: Planning under Uncertainty, Constrained Motion Planning, Collision Avoidance
Abstract: We propose a framework for planning in unknown dynamic environments with probabilistic safety guarantees using conformal prediction. Particularly, we design a model predictive controller (MPC) that uses i) trajectory predictions of the dynamic environment, and ii) prediction regions quantifying the uncertainty of the predictions. To obtain prediction regions, we use conformal prediction, a statistical tool for uncertainty quantification, that requires availability of offline trajectory data - a reasonable assumption in many applications such as autonomous driving. The prediction regions are valid, i.e., they hold with a user-defined probability, so that the MPC is provably safe. We illustrate the results in the self-driving car simulator CARLA at a pedestrian-filled intersection. The strength of our approach is compatibility with state of the art trajectory predictors, e.g., RNNs and LSTMs, while making no assumptions on the underlying trajectory-generating distribution. To the best of our knowledge, these are the first results that provide valid safety guarantees in such a setting.

Keywords: Planning under Uncertainty, Optimization and Optimal Control, Robot Safety
Abstract: Safe corridor-based Trajectory Optimization (TO) presents an appealing approach for collision-free path planning of autonomous robots, offering global optimality through its convex formulation. The safe corridor is constructed based on the perceived map, however, the non-ideal perception induces uncertainty, which is rarely considered in trajectory generation. In this paper, we propose Distributionally Robust Safe Corridor Constraints (DRSCCs) to consider the uncertainty of the safe corridor. Then, we integrate DRSCCs into the trajectory optimization framework using Bernstein basis polynomials. Theoretically, we rigorously prove that the trajectory optimization problem incorporating DRSCCs is equivalent to a computationally efficient, convex quadratic program. Compared to the nominal TO, our method enhances navigation safety by significantly reducing the infeasible motions in presence of uncertainty. Moreover, the proposed approach is validated through two robotic applications, a micro Unmanned Aerial Vehicle (UAV) and a quadruped robot Unitree A1.

Keywords: Planning under Uncertainty, Collision Avoidance, Constrained Motion Planning
Abstract: Model Predictive Path Integral (MPPI) control is a type of sampling-based model predictive control that simulates thousands of trajectories and uses these trajectories to synthesize optimal controls on-the-fly. In practice, however, MPPI encounters problems limiting its application. For instance, it has been observed that MPPI tends to make poor decisions if unmodeled dynamics or environmental disturbances exist, preventing its use in safety-critical applications. Moreover, the multi-threaded simulations used by MPPI require significant onboard computational resources, making the algorithm inaccessible to robots without modern GPUs. To alleviate these issues, we propose a novel (Shield-MPPI) algorithm that provides robustness against unpredicted disturbances and achieves real-time planning using a much smaller number of parallel simulations on regular CPUs. The novel Shield-MPPI algorithm is tested on an aggressive autonomous racing platform both in simulation and in hardware. The results show that the proposed controller greatly reduces the number of constraint violations compared to state-of-the-art robust MPPI variants and stochastic MPC methods.

Keywords: Planning under Uncertainty, Robot Safety, Collision Avoidance
Abstract: Safety is a core challenge of autonomous robot motion planning, especially in the presence of dynamic and uncertain obstacles. Many recent results use learning and deep learning-based motion planners and prediction modules to predict multiple possible obstacle trajectories and generate obstacle-aware ego robot plans. However, planners that ignore the inherent uncertainties in such predictions incur collision risks and lack formal safety guarantees. In this paper, we present a computationally efficient safety filtering solution to reduce the collision risk of ego robot motion plans using multiple samples of obstacle trajectory predictions. The proposed approach reformulates the collision avoidance problem by computing safe halfspaces based on obstacle sample trajectories using distributionally robust optimization (DRO) techniques. The safe halfspaces are used in a model predictive control (MPC)-like safety filter to apply corrections to the reference ego trajectory thereby promoting safer planning. The efficacy and computational efficiency of our approach are demonstrated through numerical simulations.

Keywords: Planning under Uncertainty, Autonomous Agents
Abstract: Real-world problems often require reasoning about hybrid beliefs, over both discrete and continuous random variables. Yet, such a setting has hardly been investigated in the context of planning. Moreover, existing

online Partially Observable Markov Decision Processes (POMDPs) solvers do not support hybrid beliefs directly. In particular, these solvers do

not address the added computational burden due to an increasing number of hypotheses with the planning horizon, which can grow exponentially. As part of this work, we present a novel algorithm, Hybrid Belief Monte

Carlo Planning (HB-MCP) that utilizes the Monte Carlo Tree Search (MCTS) algorithm to solve a POMDP while maintaining a hybrid belief. We

illustrate how the upper confidence bound (UCB) exploration bonus can be leveraged to guide the growth of hypotheses trees alongside the belief trees. We then evaluate our approach in highly aliased simulated

environments where unresolved data association leads to multi-modal belief hypotheses.

Keywords: Planning under Uncertainty, Autonomous Agents
Abstract: Autonomous agents that operate in the real world must often deal with partial observability, which is commonly modeled as partially observable Markov decision processes (POMDPs). However, traditional POMDP models rely on the assumption of complete knowledge of the observation source, known as fully observable data association. To address this limitation,

we propose a planning algorithm that maintains multiple data association hypotheses, represented as a belief mixture, where each component corresponds to a different data association hypothesis. However, this method can lead to an exponential growth in the number of hypotheses, resulting in significant computational overhead. To overcome this challenge, we introduce a pruning-based approach for planning with ambiguous data associations. Our key contribution is to derive bounds between the value function based on the complete set of hypotheses and the value function based on a pruned subset of the hypotheses, enabling us to establish a trade-off between computational efficiency and performance. We demonstrate how these bounds can both be used to certify any pruning heuristic in retrospect and propose a novel approach to determine which hypotheses to prune in order to ensure a predefined limit on the loss. We evaluate our approach in simulated environments and demonstrate its efficacy in handling multi-modal belief hypotheses with ambiguous data associations.

Keywords: Planning under Uncertainty, Formal Methods in Robotics and Automation
Abstract: Partially observable Markov decision processes (POMDPs) have been widely used in many robotic applications for sequential decision-making under uncertainty. POMDP online planning algorithms such as Partially Observable Monte-Carlo Planning (POMCP) can solve very large POMDPs with the goal of maximizing the expected return. But the resulting policies cannot provide safety guarantees which are imperative for real-world safety-critical tasks (e.g., autonomous driving). In this work, we consider safety requirements represented as almost-sure reach-avoid specifications (i.e., the probability to reach a set of goal states is one and the probability to reach a set of unsafe states is zero). We compute shields that restrict unsafe actions which would violate the almost-sure reach-avoid specifications. We then integrate these shields into the POMCP algorithm for safe POMDP online planning. We propose four distinct shielding methods, differing in how the shields are computed and integrated, including factored variants designed to improve scalability. Experimental results on a set of benchmark domains demonstrate that the proposed shielding methods successfully guarantee safety (unlike the baseline POMCP without shielding) on large POMDPs, with negligible impact on the runtime for online planning.

Keywords: Planning under Uncertainty, Motion and Path Planning
Abstract: Environments with regions of uncertain traversability can be modeled as roadmaps with probabilistic edges for efficient planning under uncertainty. We would like to generate roadmaps that enable planners to efficiently find paths with expected low costs through uncertain environments. The roadmap must be sparse so that the planning problem is tractable, but still contain edges that are likely to contribute to low-cost plans under various realizations of the environmental uncertainty. Determining the optimal set of edges to add to the roadmap without considering an exponential number of traversability scenarios is challenging. We propose the use of a heuristic that bounds the ratio between the expected path cost in our graph and the expected path cost in an optimal graph to determine whether a given edge should be added to the roadmap. We test our approach in several environments, demonstrating that our uncertainty-aware roadmaps effectively trade off between plan quality and planning efficiency for uncertainty-aware agents navigating in the graph.

Keywords: Mechanism Design, Actuation and Joint Mechanisms, Force Control
Abstract: This letter presents a design framework and novel control strategy for a compact coaxial magnetic-gear-based actuation module suitable for small-to-mid-sized mechanical and robotic applications. The proposed actuation module adopts a non-contact magnetic coupling mechanism to transmit rotational power with a predetermined gear ratio, in contrast to traditional mechanical gear-based transmissions. This approach offers several advantages such as enhanced backdrivability, hardware safety, and transparency when compared to conventional contact-based transmissions. Furthermore, the magnetic coupling effect provides a spring-like characteristic that can be utilized to implement a series elastic actuation enabling sensorless torque control. The design of the magnetic gear was optimized using a differential evolution method, and a dynamic model was formulated to specify its dynamic characteristics. Finally, a composite disturbance observer-based torque control algorithm was developed, which capitalizes on the features of the magnetic spring. The proposed control algorithm was validated through several experiments.

Keywords: Mechanism Design, Actuation and Joint Mechanisms, Underactuated Robots
Abstract: In many robotic systems, the holding state con- sumes power, limits operating time, and increases operating costs. Electrostatic clutches have the potential to improve robotic performance by generating holding torques with low power consumption. A key limitation of electrostatic clutches has been their low specific shear stresses which restrict gen- erated holding torque, limiting many applications. Here we show how combining the Johnsen-Rahbek (JR) effect with the exponential tension scaling capstan effect can produce clutches with the highest specific shear stress in the literature. Our system generated 31.3 N/cm2 sheer stress and a total holding torque of 7.1 N·m while consuming only 2.5 mW/cm2 at 500 V. We demonstrate a theoretical model of an electrostatic adhesive capstan clutch and demonstrate how large angle (θ > 2π) designs increase efficiency over planar or small angle (θ < π) clutch designs. We also report the first unfilled polymeric material, polybenzimidazole (PBI), to exhibit the JR-effect

Keywords: Mechanism Design, Biomimetics, Biologically-Inspired Robots
Abstract: This paper presents a bionic foldable wing that imitates the hind wing of ladybirds. Based on the folding mechanism of the hind wing of ladybirds and the theory of origami, the motion model of the bionic foldable wing is established, yield the motion law of the crease angles and the variation relationship between the panels are obtained. Bionic foldable wings utilise shape memory alloy to drive wings to fold, and embedded torsion springs to release energy to realize the function of wing unfolding. In the experiments of the vehicle equipped with foldable wings, the lift and attitude torque of bionic foldable wings are measured by the F/T sensor. The experimental results indicated that its aerodynamic performance is basically close to that of our optimized non-foldable wings. Moreover, the vehicle with foldable wings has been able to overcome gravity to achieve flight, which provides a novel concept for the research on flapping wing.

Keywords: Mechanism Design, Calibration and Identification, Kinematics
Abstract: Widespread applications for mobile robots are creating a large demand for developing new driving mechanisms that can handle diverse environments. Bristle-bot-like robot designs, mainly studied over the past decade, are based on vibration mechanisms built on flexible legs that enable motion on the ground. However, creating scalable and steerable bristle-bots remains a challenge. Here, we focus on developing a new kind of magnetically-driven bristle-bots with a wireless control and power supply that can be steered and downscaled.

In experiments, we verified our concept with 3D-printed bristle-bot units equipped with body-embedded permanent magnets actuated via torque imposed by an external magnetic field. An AC-powered Helmholtz coil generated the bristle-bot’s driving field, providing 2D input control, field amplitude, and frequency. A variable number of legs on each side of a bristle-bot’s body was used to ensure that each side of the bristle-bot's body has a different frequency response. This asymmetry introduces steerability by a rich set of control commands: rotations with simultaneous forward and backward locomotion. We also observed and controlled a new side locomotion phenomenon not yet described in previous studies. The results presented were supported with data from numerous experiments and thorough statistical analysis, which indicated promising directions for future developments.

Keywords: Mechanism Design, Compliance and Impedance Control, Additive Manufacturing
Abstract: Variable Stiffness Mechanisms (VSM) are becoming ubiquitous in mechatronics given the benefit they provide in terms of safety and performance. Despite these assets, VSMs remain fairly complex mechanical devices lacking in compactness, ease of manufacturing and accessibility. In addition, the scarcity of commercially available VSMs requires that such systems are mostly designed in-house. We propose a new type of VSM that improves on the pre-existing Jack Spring concept by making it more compact and robust. The new concept, which we refer to as the Compact Modifier of Active Coils (C-MAC) mechanism, is specifically designed to be manufactured through a monolithic 3D print. This approach enables to modify a minimal set of design features, namely the spring diameter and the coil diameter, to achieve the desired range of stiffness variation. We test the proposed design on six configurations; these show hysteretic energy losses no larger than 35% over the stiffness variation and confirm stiffness to scale according to theory. Stiffness ranging from 0.15 N/mm to 1.02N /mm were measured for an overall device length of 140 mm, including a maximal stroke length of 22 mm. The results confirm excellent scalability and manufacturability of the proposed design, providing a versatile mechanism for fast prototyping.

Keywords: Mechanism Design, Compliant Joints and Mechanisms, Biologically-Inspired Robots
Abstract: Plant cells expand and elongate. Their cumulative actuation defines organ morphing. Inspired by this modular transformability, this study proposes a modular concept for growing robots that will be able to grow by adding at their tip Transformable Modules (TMs). We provide a two-module implementation to evaluate the concept viability. We designed and characterized Shape-Retention Bellows (SRBs) that constitute the TM and are used to maintain the shape once the extension force is relaxed. We demonstrate module radial expansion and axial elongation in a straight and bent configuration (up to ∼4◦). This is the first concept of growing robots to enact the robot’s modularity and Transferability for future deployment in distributed growing systems capable of acting in various scenarios.

Keywords: Mechanism Design, Dynamics, Actuation and Joint Mechanisms
Abstract: This paper presents the design and experimen- tal results of a proprioceptive, high-bandwidth quasi-direct drive (QDD) actuator for highly dynamic robotic applications. A comprehensive review of the mechanical design of the PULSE115-60 actuator is presented, with particular focus on the design parameters affecting the dynamic performance of the actuator and a full specification is provided. Fundamental parameters to describe the dynamic behaviour of an actuator are discussed, and an experimental method to determine speed and torque bandwidth of the actuator is presented. A rigorous method to determine backdrive torque is also explained. Finally, experimental results quantifying the dynamic performance of the PULSE115-60 actuator are discussed. The PULSE115-60 actuator has a highly dynamic response, surpassing the torque bandwidth at low torque amplitudes showcased in state-of- the-art literature. The differences between current and torque bandwidth, two concepts often conflated in literature, are elucidated. Experimental procedures detailed in previous work are discussed and a novel standardised procedure is proposed for robust characterisation and fair comparison of different actuation systems. Finally, performance results for PULSE115- 60 are presented, demonstrating a torque bandwidth of 66.3 Hz at an amplitude of 6 N·m, ±0.11◦ of backlash and 0.37 N·m of backdrive torque.

Keywords: Mechanism Design, Engineering for Robotic Systems, Kinematics
Abstract: With the development of society, aging population and the number of stroke patients is increasing year by year. Rehabilitation exoskeleton can help patients to carry out rehabilitation training and improve their activities of daily living (ADL). First of all, a reconfigurable exoskeleton for upper limb rehabilitation is designed in this paper. The exoskeleton combines gravity compensation with left-right arm switching function through its reconfigurability. Secondly, the motion space and singular configuration of the exoskeleton are analyzed. By changing the working mode of the gravity compensation device, the control experiment of the motor is carried out. The influence of gravity compensation device on motor driving torque and energy consumption is analyzed. Finally, the results of experiment show that, in the best case, the gravity compensation device can reduce the energy consumption by 41.15% and the maximum motor current by 33.56% of the driving element.

Keywords: Mechanism Design, Flexible Robotics, Haptics and Haptic Interfaces
Abstract: A support structure for flexible displays such as OLED or flexible LEDs was developed using the flexible omnidirectional driving gear mechanism. It is a gear mechanism having two degrees of freedom on one surface. This flexible display mechanism is expected to be placed inside a car dashboard as a human interface and for workspace optimization. In this study, we propose a novel flexible omnidirectional driving gear for supporting flexible displays discussing its design, motion range, repeatability, positional accuracy, and adaptability to any guide surface through magnetic coupling. The experiments showed satisfactory results for positional accuracy and repeatability with adaptability over a wide range of curvatures.

Keywords: Formal Methods in Robotics and Automation, Force Control, Disassembly
Abstract: Flexible manufacturing lines are required to meet the demand for customized and small batch-size products. Even though state-of-the-art tactile robots may provide the versatility for increased adaptability and flexibility, their potential is yet to be fully exploited. To support robotics deployment in manufacturing, we propose a task-based tactile robot programming paradigm that uses an object-centric tactile skill definition that directly links identified object constraints of the task to the definition of constraint-based unified force-impedance control. In this study, we first explain the basic concept of abstracting the task constraints experienced by the object and transferring them to the robot's operational space frame. Second, using the object-centric tactile skill definition, we synthesize unified force-impedance control and formalized holonomic constraints to enable flexible task execution. Later, we propose the quantified analysis metrics for the process by analyzing them as a typical example of flexible manipulation disassembly skills, e.g., levering and unscrew-driving regarding their object requirements. Supported by realistic experimental evaluation using a Franka Emika robot, our tactile robot programming approach for the direct translation between task-level constraints and robot control parameter design is shown to be a viable solution for increased robotic deployment in flexible manufacturing lines.

Keywords: Formal Methods in Robotics and Automation, Hybrid Logical/Dynamical Planning and Verification
Abstract: In this paper we present a grammar and control synthesis framework for online modification of Event-based Signal Temporal Logic (STL) specifications, during execution. These modifications allow a user to change the robots' task in response to potential future violations, changes to the environment, or user-defined task design changes. In cases where a modification is not possible, we provide feedback to the user and suggest alternative modifications. We demonstrate our task modification process using a Hello Robot Stretch satisfying an Event-based STL specification.

Keywords: Formal Methods in Robotics and Automation, Hybrid Logical/Dynamical Planning and Verification, Motion and Path Planning
Abstract: This paper introduces a sampling-based strategy synthesis algorithm for nondeterministic hybrid systems with complex continuous dynamics under temporal and reachability constraints. We model the evolution of the hybrid system as a two-player game, where the nondeterminism is an adversarial player whose objective is to prevent achieving temporal and reachability goals. The aim is to synthesize a winning strategy -- a reactive (robust) strategy that guarantees the satisfaction of the goals under all possible moves of the adversarial player. The approach is based on growing a (search) game-tree in the hybrid space by combining a sampling-based planning method with a novel bandit-based technique to select and improve on partial strategies. We provide conditions under which the algorithm is probabilistically complete, i.e., if a winning strategy exists, the algorithm will almost surely find it. The case studies and benchmark results show that the algorithm is general and consistently outperforms the state of the art.

Keywords: Formal Methods in Robotics and Automation, Hybrid Logical/Dynamical Planning and Verification, Robot Safety
Abstract: In this paper, we consider the problem of safety analysis of a closed-loop control system with anytime perception sensor. We formalize the framework and present a general procedure for safety analysis using reachable set computation. We instantiate the procedure for two concrete classes, namely, the classical discrete-time linear system with linear state feedback controller and an extension with variable update rates. We present an exact computational method based on polyhedral manipulations for the first class and an over-approximate method for the second class. Our experimental results demonstrate the feasibility of the approach.

Keywords: Formal Methods in Robotics and Automation, Integrated Planning and Learning, Model Learning for Control
Abstract: The robustness of signal temporal logic not only assesses whether a signal adheres to a specification but also provides a measure of how much a formula is fulfilled or violated. The calculation of robustness is based on evaluating the robustness of underlying predicates. However, the robustness of predicates is usually defined in a model-free way, i.e., without including the system dynamics. Moreover, it is often nontrivial to define the robustness of complicated predicates precisely. To address these issues, we propose a notion of model predictive robustness, which provides a more systematic way of evaluating robustness compared to previous approaches by considering model-based predictions. In particular, we use Gaussian process regression to learn the robustness based on precomputed predictions so that robustness values can be efficiently computed online. We evaluate our approach for the use case of autonomous driving with predicates used in formalized traffic rules on a recorded dataset, which highlights the advantage of our approach compared to traditional approaches in terms of expressiveness. By incorporating our robustness definitions into a trajectory planner, autonomous vehicles obey traffic rules more robustly than human drivers in the dataset.

Keywords: Formal Methods in Robotics and Automation, Learning from Demonstration, Probability and Statistical Methods
Abstract: In the realm of robotics, numerous downstream robotics tasks leverage machine learning methods for processing, modeling, or synthesizing data. Often, this data comprises variables that inherently carry geometric constraints, such as the unit-norm condition of quaternions representing rigid-body orientations or the positive definiteness of stiffness and manipulability ellipsoids. Handling such geometric constraints effectively requires the incorporation of tools from differential geometry into the formulation of machine learning methods. In this context, Riemannian manifolds emerge as a powerful mathematical framework to handle such geometric constraints. Nevertheless, their recent adoption in robot learning has been largely characterized by a mathematically-flawed simplification, hereinafter referred to as the "single tangent space fallacy". This approach involves merely projecting the data of interest onto a single tangent (Euclidean) space, over which an off-the-shelf learning algorithm is applied. This paper provides a theoretical elucidation of various misconceptions surrounding this approach and offers experimental evidence of its shortcomings. Finally, it presents valuable insights to promote best practices when employing Riemannian geometry within robot learning applications.

Keywords: Formal Methods in Robotics and Automation, Multi-Robot Systems
Abstract: In this work, we address the problem of control synthesis for a homogeneous team of robots given a global temporal logic specification and formal user preferences for relaxation in case of infeasibility. The relaxation preferences are represented as a Weighted Finite-state Edit System and are used to compute a relaxed specification automaton that captures all allowable relaxations of the mission specification and their costs. For synthesis, we introduce a Mixed Integer Linear Programming (MILP) formulation that combines the motion of the team of robots with the relaxed specification automaton. Our approach combines automata-based and MILP-based methods and leverages the strengths of both approaches, while avoiding their shortcomings. Specifically, the relaxed specification automaton explicitly accounts for the progress towards satisfaction, and the MILP-based optimization approach avoids the state-space explosion associated with explicit product-automata construction, thereby efficiently solving the problem. The case studies highlight the efficiency of the proposed approach

Keywords: Formal Methods in Robotics and Automation, Multi-Robot Systems
Abstract: This paper introduces an iterative approach to multi-agent route planning under chance constraints. A heterogeneous team of agents with various capabilities is tasked with a Capability Temporal Logic (CaTL) mission, a fragment of Signal Temporal Logic. The agents' motion is modeled as a finite weighted graph, where the weights represent travel durations. Given the probability distribution over the durations of each edge's traversal, we want to find paths for all agents such that (a) the specification robustness is maximized, (b) travel time is minimized, and (c) the success probability is maximized. We tackle the problem using an iterative approach. In each stage, it selects edges' traversal duration and success probabilities and then solves a multi-agent route planning problem. We use an efficient Mixed-Integer Linear Programming (MILP) encoding for the latter. Our method provides a framework for agents to make informed decisions in choosing the most suitable edge attributes (travel durations and success probabilities) that consider agents' capabilities to perform tasks in the environment. The proposed iterative method leverages graph structure to generate a more efficient search space. The effectiveness of our method is demonstrated through simulated case studies where obtaining the optimal solution would otherwise be computationally expensive. Our approach efficiently explores the solution space, generating better solutions and improving the performance of multi-agent route planning with uncertain travel durations.

Keywords: Formal Methods in Robotics and Automation, Networked Robots, Telerobotics and Teleoperation
Abstract: Autonomous robots must utilize rich sensory data to make safe control decisions. To process this data, compute-constrained robots often require assistance from remote computation, or the cloud, that runs compute-intensive deep neural network perception or control models. However, this assistance comes at the cost of a time delay due to network latency, resulting in past observations being used in the cloud to compute the control commands for the present robot state. Such communication delays could potentially lead to the violation of essential safety properties, such as collision avoidance. This paper develops methods to ensure the safety of robots operated over communication networks with stochastic latency. To do so, we use tools from formal verification to construct a shield, i.e., a run-time monitor, that provides a list of safe actions for any delayed sensory observation, given the expected and maximum network latency. Our shield is minimally intrusive and enables networked robots to satisfy key safety constraints, expressed as temporal logic specifications, with desired probability. We demonstrate our approach on a real F1/10th autonomous vehicle that navigates in indoor environments and transmits rich LiDAR sensory data over congested WiFi links.

Keywords: Reinforcement Learning, Aerial Systems: Perception and Autonomy, Aerial Systems: Applications
Abstract: Looking back at previous diversity work on Rein-forced learning, diversity is often achieved through the augmented loss function, which is required in the context of reward and diversity. Usually, the diversity optimization algorithm uses the multi-armed bandit algorithm to select the coefficient that predefined space. However, the dynamic distribution of the reward signal or quality of the MAB with diversity limits the performance of these methods. We introduce the Phase Diversity Optimization (PDO) algorithm, a population-based training-based framework that combines reward and diversity training to different stages, rather than optimizing multi-objective functions. In the secondary phase, having poor performance diversification through determinants does not replace the better agents in the archive. Rewards and diversity allow us to use the diversity of positive optimization in the secondary phase, where performance does not degrade. Furthermore, we built an aerial melee scenario agent.

Keywords: Reinforcement Learning, Aerial Systems: Perception and Autonomy, Vision-Based Navigation
Abstract: This work is focused on reinforcement learning (RL)-based navigation for drones, whose localisation is based on visual odometry (VO). Such drones should avoid flying into areas with poor visual features, as this can lead to deteriorated localization or complete loss of tracking. To achieve this, we propose a hierarchical control scheme, which uses an RL-trained policy as the high-level controller to generate waypoints for the next control step and a low-level controller to guide the drone to reach subsequent waypoints. For the high-level policy training, unlike other RL-based navigation approaches, we incorporate awareness of VO performance into our policy by introducing pose estimation-related punishment. To aid robots in distinguishing between perception-friendly areas and unfavoured zones, we instead provide semantic scenes, as input for decision-making instead of raw images. This approach also helps minimise the sim-to-real application gap.

Keywords: Multi-Robot Systems, Deep Learning Methods, Human-Robot Collaboration
Abstract: We propose a novel approach to multi-robot collaboration that harnesses the power of pre-trained large language models (LLMs) for both high-level communication and low-level path planning. Robots are equipped with LLMs to discuss and collectively reason task strategies. They generate sub-task plans and task space waypoint paths, which are used by a multi-arm motion planner to accelerate trajectory planning. We also provide feedback from the environment, such as collision checking, and prompt the LLM agents to improve their plan and waypoints in-context. For evaluation, we introduce RoCoBench, a 6-task benchmark covering a wide range of multi-robot collaboration scenarios, accompanied by a text-only dataset that evaluates LLMs’ agent representation and reasoning capability. We experimentally demonstrate the effectiveness of our approach — it achieves high success rates across all tasks in RoCoBench and adapts to variations in task semantics. Our dialog setup offers high interpretability and flexibility — in real world experiments, we show RoCo easily incorporates human-in-the-loop, where a user can communicate and collaborate with a robot agent to complete tasks together. Project website: https://project-roco.github.io

Keywords: Multi-Robot Systems, Reinforcement Learning, Collision Avoidance
Abstract: End-to-end deep reinforcement learning (DRL) for quadrotor control promises many benefits -- easy deployment, task generalization and real-time execution capability. Prior end-to-end DRL-based methods have showcased the ability to deploy learned controllers onto single quadrotors or quadrotor teams maneuvering in simple, obstacle-free environments. However, the addition of obstacles increases the number of possible interactions exponentially, thereby increasing the difficulty of training RL policies. In this work, we propose an end-to-end DRL approach to control quadrotor swarms in environments with obstacles. We provide our agents a curriculum and a replay buffer of the clipped collision episodes to improve performance in obstacle-rich environments. We implement an attention mechanism to attend to the neighbor robots and obstacle interactions - the first successful demonstration of this mechanism on policies for swarm behavior deployed on severely compute-constrained hardware. Our work is the first work that demonstrates the possibility of learning neighbor-avoiding and obstacle-avoiding control policies trained with end-to-end DRL that transfers zero-shot to real quadrotors. Our approach scales to 32 robots with 80% obstacle density in simulation and 8 robots with 20% obstacle density in physical deployment. Website: https://sites.google.com/view/obst-avoid-swarm-rl.

Keywords: Multi-Robot Systems, Reinforcement Learning, Distributed Robot Systems
Abstract: Deep reinforcement learning (DRL) methods have been widely applied in distributed multi-robotic systems and successfully realized autonomous learning in many fields. In these fields, robots need to communicate and collaborate with other robots in real time, and reach agreed cognition for task assignment, which puts high requirements on efficiency and stability. However, robots may often get damaged even crash in complex environments, and have to be dynamically substituted. It seems not robust enough for most existing DRL works to make new robots fast adapt to current team policies, causing performance degradation. In this work, we get inspired by the genetic mechanism of social animals' instincts, and propose a robust multi-robotic collaboration and communication framework, C3F. It introduces graph-based representation to discover more features on the relevance among robots, and takes advantage of meta learning mechanism to conclude the general meta policy. When some robots crash and get replaced by new ones, this meta policy will be reused to guide new robots on how to quickly follow the existing collaboration and communication rules, and fast adapt to their roles in the team. The experiments on both the Webots simulator and the Starcraft II platform indicate that our methods have better performance compared with some SOTA methods, showing strong robustness and remarkable adaptability to the dynamic substitution in multi-robotic systems.

Keywords: Multi-Robot Systems, Reinforcement Learning, Optimization and Optimal Control
Abstract: Rollout algorithms are renowned for their abilities to correct for the suboptimalities of offline-trained base policies. In the multiagent setting, performing online rollout can require an exponentially large number of optimizations with respect to the number of agents. One-agent-at-a-time algorithms offer computationally efficient approaches to guaranteed policy improvement; however, this improvement is with respect to a state value estimate derived from a potentially poor base policy. Monte Carlo tree search (MCTS) provably converges to the true state value estimates; however, the exponentially large search space often makes its online use limited. Here, we present the Multi-Level Action Tree Rollout (MLAT-R) algorithm. MLAT-R provides 1) provable improvement over a base policy, 2) policy improvement with respect to the true state value, 3) applicability to any number of agents, and 4) an action space that grows linearly with the number of agents rather than exponentially. In this paper, we outline the algorithm, sketch a proof of its improvement over a base policy, and evaluate its performance on a challenging problem for which the base policy cannot reach a terminal state. Despite the challenging experimental setup, our algorithm reached a terminal state in 86% of all experiments, compared to 31% for state-of-the-art one-agent-at-a-time algorithms. In experiments involving MCTS, MLAT-R reached a terminal state in 99% of experiments compared to 92% for MCTS. MLAT-R achieved these results while considering an exponentially smaller action space than MCTS.

Keywords: Multi-Robot Systems, Reinforcement Learning, Transfer Learning
Abstract: It is common for us to feel pressure in a competition environment, which arises from the desire to obtain success comparing with other individuals or opponents. Although we might get anxious under the pressure, it could also be a drive for us to stimulate our potentials to the best in order to keep up with others. Inspired by this, we propose a competitive learning framework which is able to help individual robot to acquire knowledge from the competition, fully stimulating its dynamics potential in the race. Specifically, the competition information among competitors is introduced as the additional auxiliary signal to learn advantaged actions. We further build a Multiagent-Race environment, and extensive experiments are conducted, demonstrating that robots trained in competitive environments outperform ones that are trained with SoTA algorithms in single robot environment.

Keywords: Multi-Robot Systems, Vision-Based Navigation, Reinforcement Learning
Abstract: Communication is a key element in applying multi-agent reinforcement learning to a wide range of real-world scenarios. We focus on optical wireless communication (OWC), which is a practical solution to be used in various real situations where radio communication is not available, such as underwater or in a lot of radio noise environment. OWC is a method of communicating only with other agents in visual range using light, unlike radio wave like communication which is mostly assumed in existing research on multi-agent reinforcement learning. Due to limited communication, when OWC is used, overall performance is generally degraded from the case with full communication. In this paper, we propose a reinforcement learning method that learns visual coordination behavior using OWC. Our proposed visually cooperative behavior enables agents equipped with limited field of view (FOV) cameras to efficiently comprehend and imagine their surrounding environment through cooperative communication. Experimental results in simulation demonstrated that, using the proposed visual coordination method, multi-agents using OWC with general FOV show comparable performance to those with radio wave like full communication. Additionally, it has been demonstrated that this method can improve performance in various multi-agent reinforcement learning algorithms. We also implement OWC devices on real mobile robots and demonstrated the proposed multi-agent operation.

Keywords: Sensor Fusion, Human Detection and Tracking, Legged Robots
Abstract: Tracking a specific person in 3D scene is gaining momentum due to its numerous applications in robotics. Currently, most 3D trackers focus on driving scenarios with neglected jitter and uncomplicated surroundings, which results in their severe degeneration in complex environments, especially on jolting robot platforms (only 20-60% success rate). To improve the accuracy, a Point-Video-based Transformer Tracking model (PVTrack) is presented for robots. It is the first multi-modal 3D human tracking work that incorporates point clouds together with RGB videos to achieve information complementarity. Moreover, PVTrack proposes the Siamese Point-Video Transformer for feature aggregation to overcome dynamic environments, which captures more target-aware information through the hierarchical attention mechanism adaptively. Considering the violent shaking on robots and rugged terrains, a lateral Human-ware Proposal Network is designed together with an Anti-shake Proposal Compensation module. It alleviates the disturbance caused by complex scenes as well as the particularity of the robot platform. Experiments show that our method achieves state-of-the-art performance on both KITTI/Waymo datasets and a quadruped robot for various indoor and outdoor scenes.

Keywords: Sensor Fusion, Neurorobotics, Multi-Modal Perception for HRI
Abstract: As intelligent systems become increasingly important in our daily lives, new ways of interaction are needed. Classical user interfaces pose issues for the physically impaired and are partially not practical or convenient. Gesture recognition is an alternative, but often not reactive enough when conventional cameras are used. This work proposes a Spiking Convolutional Neural Network, processing event- and depth data for gesture recognition. The network is simulated using the open-source neuromorphic computing framework LAVA for offline training and evaluation on an embedded system. For the evaluation three open source data sets are used. Since these do not represent the applied bi-modality, a new data set with synchronized event- and depth data was recorded. The results show the viability of temporal encoding on depth information and modality fusion, even on differently encoded data, to be beneficial to network performance and generalization capabilities.

Keywords: Sensor-based Control, Optical Proximity Sensor, Reactive and Sensor-Based Planning, Physical Human-Robot Interaction
Abstract: This study proposes novel control methods that lower impact force by preemptive movement and smoothly transition to conventional contact-based impedance control. These techniques are suggested for application in force control-based robots and position/velocity control-based robots. Strong impact forces have a negative influence on multiple robotic tasks. Recently, preemptive impact reduction techniques that expand conventional contact impedance control using proximity sensors have been examined. However, a seamless transition from impact reduction to contact impedance control has yet to be demonstrated. In contrast, our proposed methods solve this problem. The preemptive impact reduction feature can be added to an already-implemented impedance controller because the parameter design is divided into impact reduction and contact impedance control. There is no abrupt alteration in the contact force during the transition. Furthermore, although the preemptive impact reduction uses a crude optical proximity sensor, the influence of reflectance is minimized. Analyses and real-world experiments confirm these features, which are useful for many robots performing contact tasks.

Keywords: Sensor-based Control, Robot Safety, Aerial Systems: Perception and Autonomy
Abstract: Control barrier functions have become an increasingly popular framework for safe real-time control. In this work, we present a computationally low-cost framework for synthesizing barrier functions over point cloud data for safe vision-based control. We take advantage of surface geometry to locally define and synthesize a quadratic CBF over a point cloud. This CBF is used in a CBF-QP for control and verified in simulation on quadrotors and in hardware on quadrotors and the TurtleBot3. This technique enables safe navigation through unstructured and dynamically changing environments and is shown to be significantly more efficient than current methods.

Keywords: Sensor-based Control, Robust/Adaptive Control
Abstract: In this paper, we discuss the stability analysis of Plane-to-Plane positioning task when the task is designed in proximity sensor space. We utilize a multi-sensor arrangement of proximity sensors that forms a proximity array to obtain the necessary information in sensor space. For the task considered, we provide closed-form equations for the closed-loop system by obtaining the analytical form of pseudo-inverse for the interaction matrix involved. This further enables us to suggest a new control law producing a decoupled exponential decrease of the sensor errors in perfect conditions, while being more robust to estimation errors in the surface normal. By applying Gershgorin’s theorem to the closed-form matrices, we are able to provide conditions for stability with respect to errors in extrinsic parameters and surface normal. Simulation results are provided to discuss the robustness of the task with respect to these modeling parameters.

Keywords: Sensor-based Control, Vision-Based Navigation, SLAM
Abstract: Visual odometry system is challenged by complex illumination environments. Image quality and its consistency in the time domain directly determine feature detection and tracking performance, which further affect the robustness and accuracy of the entire system. In this paper, an image acquisition scheme with image bracketing patterns is proposed. Images with different exposure levels are continuously captured to explore the scene under varying illumination sufficiently. An attribute control method is designed to adjust image exposures within the brackets online. Gaussian process regression fits the relationship between image quality metric and exposure via image synthesis technique, and the optimal exposures for the next bracket are obtained directly without attempts to ensure a quick response. Experiments show our acquisition system's effectiveness and performance improvement for VO tasks in complex illumination scenes.

Keywords: Additive Manufacturing
Abstract: Accurate robotic control over interactions with the environment is fundamentally grounded in understanding tactile contacts. In this paper, we introduce MagicTac, a novel high-resolution grid-based tactile sensor. This sensor employs a 3D multi-layer grid-based design, inspired by the Magic Cube structure. This structure can help increase the spatial resolution of MagicTac to perceive external interaction contacts. Moreover, the sensor is produced using the multi-material additive manufacturing technique, which simplifies the manufacturing process while ensuring repeatability of production. Compared to traditional vision-based tactile sensors, it offers the advantages of i) high spatial resolution, ii) significant affordability, and iii) fabrication-friendly construction that requires minimal assembly skills. We evaluated the proposed MagicTac in the tactile reconstruction task using the deformation field and optical flow. Results indicated that MagicTac could capture fine textures and is sensitive to dynamic contact information. Through the grid-based multi-material additive manufacturing technique, the affordability and productivity of MagicTac can be enhanced with a minimum manufacturing cost of £4.76 and a minimum manufacturing time of 24.6 minutes.

Keywords: Robot Audition, Localization
Abstract: We present a novel sound source localization method that leverages microphone pair training, designed to deliver robust performance in various real-world environments. Existing deep learning (DL)-based approaches face scalability issues when dealing with various types of microphone arrays. To address these issues, our approach has been structured into two training steps: the first step focuses on microphone pair training, while the second step is designed for array geometry-aware training. The first training step enables our model to learn from multiple datasets covering various real-world situations, allowing it to robustly estimate the time difference of arrival (TDoA). Our robust-TDoA model incorporates a Mel scale learnable filter bank (MLFB) and a hierarchical frequency-to-time attention network (HiFTA-net). This allows it to effectively learn from various situations in multiple datasets, including those involving simultaneous sources and various sound events. The second training step enables our approach to estimate the direction of arrival (DoA) of sound based on TDoA information computed by our robust-TDoA model, which begins with parameters acquired during the first training step. During this process, our approach can be trained to accommodate geometry information of the target microphone array, which can span diverse array types. As a result, our method demonstrates robust performance across two DoA estimation tasks using three different types of arrays.

Keywords: Robot Audition, Software-Hardware Integration for Robot Systems, Physical Human-Robot Interaction
Abstract: In the last few years, mobile robots such as floor cleaners, assistive robots, and home telepresence have become an essential part of our day-to-day activities. In human-computer interaction, speech is the preferred way of communication, especially in indoor environments. This paper proposes a speech module to rotate the mobile robot. It has two components, namely, a distant automatic speech recognizer and a sound source localizer. To build distant speech recognizer, far-field speech data is collected at 1, 3, and 5-meter distances. The model performs well even at a 5-meter distance with a Word Error Rate of 40.38% and a Character Error Rate of 28.85%. The direction of arrival of the speech signal is computed from the 4-mic circular array microphone. The speech module is integrated with the Robot Operating System and physically demonstrated on Turtlebot3 Waffle Pi. It is observed that the speech recognizer and sound source localizer work well in the reverberant indoor environment with a small single-board computer.

Keywords: Recognition, Computer Vision for Automation, Transfer Learning
Abstract: When used in a real-world noisy environment, the capacity to generalize to multiple domains is essential for any autonomous scene text spotting system. However, existing state-of-the-art methods employ pretraining and fine-tuning strategies on natural scene datasets, which do not exploit the feature interaction across other complex domains. In this work, we explore and investigate the problem of domain-agnostic scene text spotting, i.e., training a model on multi-domain source data such that it can directly generalize to target domains rather than being specialized for a specific domain or scenario. In this regard, we present the community a text spotting validation benchmark called Under-Water Text (UWT) for noisy underwater scenes to establish an important case study. Moreover, we also design an efficient super-resolution based end-to-end transformer baseline called DA-TextSpotter which achieves comparable or superior performance over existing text spotting architectures for both regular and arbitrary-shaped scene text spotting benchmarks in terms of both accuracy and model efficiency. The dataset, code and pre-trained models have been released in our GitHub.

Keywords: Recognition, Deep Learning for Visual Perception, Visual Learning
Abstract: Point cloud is a popular and widely used geometric representation, which has attracted significant attention in 3D vision. However, the geometric variability of point cloud representations across different datasets can cause domain discrepancies, which hinder knowledge transfer and model generalization, resulting in degraded performance in target domain. In this paper, we present a novel approach to improve point cloud domain adaptation by employing masked representation learning in a self-supervised manner. Specifically, our method combines masked feature prediction and masked sample consistency to encode both local structure and global semantic information for learning invariant point cloud representation across domains. Moreover, to learn domain-specific representation and transfer knowledge from source to target, we propose prototype-calibrated self-training. By exploiting class-wise prototypes in the shared feature space, the soft pseudo labels can be adaptively denoised, which benefits the decision boundary learning in target domain. We conduct experiments on PointDA-10 and PointSegDA for 3D point cloud shape classification and semantic segmentation, respectively. The results demonstrate the effectiveness of our method and show that we can achieve the new state-of-the-art performance on point cloud domain adaptation.

Keywords: Recognition, Human Detection and Tracking, AI-Enabled Robotics
Abstract: In scenarios of Human-Robot Interaction (HRI), it is often assumed that the robot should cooperate with the closest individual or that only one person is present. However, in real-life situations, such as shop floor operations, this assumption may not hold. Thus, it becomes necessary for a robot to recognize a specific target in a crowded environment. To address this problem, we propose a person re-identification module that uses continuous visual adaptation techniques. This module ensures that the robot can seamlessly cooperate with the appropriate individual despite its appearance changes or partial or total occlusions. We used both a laboratory environment and an HRI scenario where the robot followed a person to test our framework. During the test, the targets were asked to change their appearance and disappear from the camera's field of view to test the module's ability to handle challenging cases of occlusion and outfit variations. We compared our framework with a state-of-the-art Multi-Object Tracking (MOT) method, and the results showed that our module, shortly named CARPE-ID, accurately tracked each selected target throughout the experiments in all cases except for two cases. In contrast, the MOT had an average of 4 tracking errors for each video.

Keywords: Recognition, Localization, Computer Vision for Automation
Abstract: In large-scale metric localization, an incorrect result during retrieval will lead to an incorrect pose estimate or loop closure. Re-ranking methods propose to take into account all the top retrieval candidates and re-order them to increase the likelihood of the top candidate being correct. However, state-of-the-art re-ranking methods are inefficient when re-ranking many potential candidates due to their need for resource intensive point cloud registration between the query and each candidate. In this work, we propose an efficient spectral method for geometric verification (named SpectralGV) that does not require registration. We demonstrate how the optimal inter-cluster score of the correspondence compatibility graph of two point clouds represents a robust fitness score measuring their spatial consistency. This score takes into account the subtle geometric differences between structurally similar point clouds and therefore can be used to identify the correct candidate among potential matches retrieved by global similarity search. SpectralGV is deterministic, robust to outlier correspondences, and can be computed in parallel for all potential candidates. We conduct extensive experiments on 5 large-scale datasets to demonstrate that SpectralGV outperforms other state-of-the-art re-ranking methods and show that it consistently improves the recall and pose estimation of 3 state-of-the-art metric localization architectures while having a negligible effect on their runtime.

Keywords: Recognition, Semantic Scene Understanding, Multi-Modal Perception for HRI
Abstract: This paper presents Action-SGFA, a novel action feature alignment approach to learn unified joint embeddings across four action modalities incorporating scene graph (SG) comprehension. A new training paradigm for Action-SGFA is also devised to improve pre-trained VL models using datasets with SG annotation. When learning from image-SG pairs, it captures structure-associated action knowledge for visual and textual encoders. SG supervision generates fine-grained captions based on various graph augmentations highlighting different compositional aspects of action scenes. Furthermore, our research reveals that all combinations of paired data are unnecessary to train such unified embeddings, and only image-paired data is sufficient to bind all action modalities together. Our Action-SGFA can leverage existing large VL models, enhancing their zero-shot capabilities of new modalities due to their natural pairings with images. The open-vocabulary zero-shot performance improves with the strength of the pre-trained VL model and the SG comprehension. We establish a new state-of-the-art in several zero-shot action recognition tasks across modalities, significantly surpassing the vanilla skeleton zero-shot method by 27.0% and 19.7% on NTU-60 and NTU-120, respectively. Additionally, in the context of RGB videos, we surpass the state-of-the-art method on Kinetics-400 by 2.1%.

Keywords: Recognition, Vision-Based Navigation, Computer Vision for Transportation
Abstract: With innovation in fields such as autonomous driving and augmented reality, point cloud-based place recognition has gained significant attention. Many methods try to address this problem by extracting and matching global descriptors in a database, but they often must balance the extraction of comprehensive contextual information and large model sizes. To overcome this challenge, we propose a lightweight parameter-shared network (LPS-Net), which includes multiple bidirectional perception units (BPUs) to extract multi-scale long-range contextual information and parameter-shared NetVLADs (PS-VLADs) to aggregate descriptors. A BPU includes a parameter-shared convolution module (SharedConv) that significantly compresses the model and enhances its ability to capture informative features. In PS-VLADs, we replace half the parameters used in the original NetVLAD with trainable scalars, which further reduces the model size, and theoretically prove their equivalence. Experimental results demonstrate that LPS-Net achieves state-of-the-art performance at the task of point cloud-based place recognition while maintaining a small model size. Code and supplementary materials can be found at https://github.com/Yavinr/LPS-Net.

Keywords: Visual Tracking, Computer Vision for Automation, Aerial Systems: Perception and Autonomy
Abstract: Correlation filter (CF)-based approaches have gained widespread attention in the field of unmanned aerial vehicle (UAV) visual tracking due to their light-weight characteristics. However, CFs are prone to generating low-quality response in challenging UAV scenarios, e.g., fast motion and background clutter. In this paper, in order to model the tracker more robustly, we first conduct an effective regularization analysis from the perspectives of response- and background-learning. Specifically, to address response degradation, we propose a module for learning temporal consistency and reversibility of response, supplemented by a novel background-aware module to enhance the ability to learn from negative samples. In addition, we propose a fast coarse-to-fine scale search strategy, which alleviates the challenges in estimating bounding boxes under non-uniform aspect ratios. We have developed two tracker versions, namely RBLT and DeepRBLT, based on the depth of the features. Comprehensive experiments on four UAV benchmarks and one generic benchmark have indicated the superiority of our trackers compared to other state-of-the-art trackers, with enough speed for real-time applications.

Keywords: Deep Learning for Visual Perception, Recognition, Visual Learning
Abstract: As robotic systems increasingly encounter complex and unconstrained real-world scenarios, there is a demand to recognize diverse objects. The state-of-the-art 6D object pose estimation methods rely on object-specific training and therefore do not generalize to unseen objects. Recent novel object pose estimation methods are solving this issue using task-specific fine-tuned CNNs for deep template matching. This adaptation for pose estimation still requires expensive data rendering and training procedures. MegaPose for example is trained on a dataset consisting of two million images showing 20,000 different objects to reach such generalization capabilities. To overcome this shortcoming we introduce ZS6D, for zero-shot novel object 6D pose estimation. Visual descriptors, extracted using pre-trained Vision Transformers (ViT), are used for matching rendered templates against query images of objects and for establishing local correspondences. These local correspondences enable deriving geometric correspondences and are used for estimating the object's 6D pose with RANSAC-based Ptextit{n}P. This approach showcases that the image descriptors extracted by pre-trained ViTs are well-suited to achieve a notable improvement over two state-of-the-art novel object 6D pose estimation methods, without the need for task-specific fine-tuning. Experiments are performed on LMO, YCBV, and TLESS. In comparison to MegaPose, we improve the Average Recall on all three datasets and compared to OSOP we improve on two datasets. The code is available at https://github.com/PhilippAuss/ZS6D.

Keywords: Recognition, Deep Learning for Visual Perception, Computer Vision for Automation
Abstract: Vision transformers have demonstrated impressive performance in various robotics and automation applications, such as classification automation and action recognition. However, the drawback of transformers is their quadratic increase in computing resources with larger inputs and dependence on considerable data for training. Most action recognition models using the transformer structure rely on a few frames from the original video to reduce computation, so temporal information is compromised by low frame rates. Spatial information is also compromised by reducing the number of embeddings as the transformer layer iterates. The letter proposes a robust model for action recognition that overcomes the limitations of most action recognition models with the transformer structure using the duplex attention function, flow-guided information, RGB information, and spatial support tokens. The proposed duplex attention mechanism leverages optical flow and RGB to address the lack of temporal information. The method employs spatial interest clustering to convert input data into tokens, improving the preservation of spatial information. Finally, meaningful action event frames are extracted by analyzing the flow and clustering to distinguish scenes. The experimental results reveal that the proposed model outperforms state-of-the-art methods in action recognition accuracy.

Keywords: Reinforcement Learning, Task and Motion Planning, Continual Learning
Abstract: To assist with everyday human activities, robots must solve complex long-horizon tasks and generalize to new settings. Recent deep reinforcement learning (RL) methods show promise in fully autonomous learning, but they struggle to reach long-term goals in large environments. On the other hand, Task and Motion Planning (TAMP) approaches excel at solving and generalizing across long-horizon tasks, thanks to their powerful state and action abstractions. But they assume predefined skill sets, which limits their real-world applications. In this work, we combine the benefits of these two paradigms and propose an integrated task planning and skill learning framework named LEAGUE (Learning and Abstraction with Guidance). LEAGUE leverages the symbolic interface of a task planner to guide RL- based skill learning and creates abstract state space to enable skill reuse. More importantly, LEAGUE learns manipulation skills in-situ of the task planning system, continuously growing its capability and the set of tasks that it can solve. We evaluate LEAGUE on four challenging simulated task domains and show that LEAGUE outperforms baselines by large margins. We also show that the learned skills can be reused to accelerate learning in new tasks domains and transfer to a physical robot platform.

Keywords: Continual Learning, Deep Learning for Visual Perception, Computer Vision for Transportation
Abstract: Prior to the deployment of robotic systems, pre-training the deep-recognition models on all potential visual cases is infeasible in practice. Hence, test-time adaptation (TTA) allows the model to adapt itself to novel environments and improve its performance during test time (i.e., lifelong adaptation). Several works for TTA have shown promising adaptation performances in continuously changing environments. However, our investigation reveals that existing methods are vulnerable to dynamic distributional changes and often lead to overfitting of TTA models. To address this problem, this paper first presents a robust TTA framework with compound domain knowledge management. Our framework helps the TTA model to harvest the knowledge of multiple representative domains (i.e., compound domain) and conduct the TTA based on the compound domain knowledge. In addition, to prevent overfitting of the TTA model, we devise novel regularization which modulates the adaptation rates using domain-similarity between the source and the current target domain. With the synergy of the proposed framework and regularization, we achieve consistent performance improvements in diverse TTA scenarios, especially on dynamic domain shifts. We demonstrate the generality of proposals via extensive experiments including image classification on ImageNet-C and semantic segmentation on GTA5, C-driving, and corrupted Cityscapes datasets.

Keywords: Continual Learning, Incremental Learning, Deep Learning for Visual Perception
Abstract: Lifelong learning or continual learning is the problem of training an AI agent continuously while also preventing it from forgetting its previously acquired knowledge. Streaming lifelong learning is a challenging setting of lifelong learning with the goal of continuous learning in a dynamic non-stationary environment without forgetting. We introduce a novel approach to lifelong learning, which is streaming (observes each training example only once), requires a single pass over the data, can learn in a class-incremental manner, and can be evaluated on-the-fly (anytime inference). To accomplish these, we propose a novel emph{virtual gradients} based approach for continual representation learning which adapts to each new example while also generalizing well on past data to prevent catastrophic forgetting. Our approach also leverages an exponential-moving-average-based semantic memory to further enhance performance. Experiments on diverse datasets with temporally correlated observations demonstrate our method's efficacy and superior performance over existing methods.

Keywords: Continual Learning, Reinforcement Learning, Incremental Learning
Abstract: Continual reinforcement learning, which aims to help robots acquire skills without catastrophic forgetting, obviating the need to re-learn all tasks from scratch. In order to enable lifelong acquisition of skills in robots, replay-based continual reinforcement learning has emerged as a promising research direction. These techniques replay data from previous tasks to mitigate forgetting when learning new skills. However, existing replay-based methods store poor representative experience, and the experience utilization of old tasks is inefficient. To address these issues, we propose a experience consistency distillation method for robot continual reinforcement learning to improve the data efficiency of the experience. Specifically, the experience of old tasks are distilled to obtain Markov Decision Process (MDP) data with high compression ratio and information content. To ensure consistent data distributions before and after distillation, we further utilize a Fréchet Inception Distance (FID) loss as a regularization constraint. In order to improve experience utilization efficiency, the policy is then trained using both the distilled data and current task data, with policy distillation performed based on uncertainty metrics. Our method is validated in the continual reinforcement learning simulation platform and real scene with a UR5e robot arm. Experimental results indicate that our method achieves higher success and lower buffer size requirement compared to other methods.

Keywords: Integrated Planning and Learning, Continual Learning, Reinforcement Learning
Abstract: Open-world robotic tasks such as autonomous driving pose significant challenges to robot control due to unknown and unpredictable events that disrupt task performance. Neural network-based reinforcement learning (RL) techniques (like DQN, PPO, SAC, etc.) struggle to adapt in large domains and suffer from catastrophic forgetting. Hybrid planning and RL approaches have shown some promise in handling environmental changes but lack efficiency in accommodation speed. To address this limitation, we propose an enhanced hybrid system with a nested hierarchical action abstraction that can utilize previously acquired skills to effectively tackle unexpected novelties. We show that it can adapt faster and generalize better compared to state-of-the-art RL and hybrid approaches, significantly improving robustness when multiple environmental changes occur at the same time.

Keywords: Continual Learning, Learning from Experience, Learning from Demonstration
Abstract: Large Language Models (LLMs) have emerged as a new paradigm for embodied reasoning and control, most recently by generating robot policy code that utilizes a custom library of vision and control primitive skills. However, prior arts fix their skills library and steer the LLM with carefully hand-crafted prompt engineering, limiting the agent to a stationary range of addressable tasks. In this work, we introduce LRLL, an LLM-based lifelong learning agent that continuously grows the robot skill library to tackle manipulation tasks of ever-growing complexity. LRLL achieves this with four novel contributions : 1) a soft memory module that allows dynamic storage and retrieval of past experiences to serve as context, 2) a self-guided exploration policy that proposes new tasks in simulation, 3) a skill abstractor that distills recent experiences into new library skills, and 4) a lifelong learning algorithm for enabling human users to bootstrap new skills with minimal online interaction. LRLL continuously transfers knowledge from the memory to the library, building composable, general and interpretable policies, while bypassing gradient-based optimization, thus relieving the learner from catastrophic forgetting. Empirical evaluation in a simulated tabletop environment shows that LRLL outperforms end-to-end and vanilla LLM approaches in the lifelong setup, while learning skills that are transferable to the real world. Project material will become available at the webpage https://gtziafas.github.io/LRLL_project/

Keywords: Continual Learning, Manipulation Planning, Integrated Planning and Learning
Abstract: Large Language Models (LLMs) have been shown to act like planners that can decompose high-level instructions into a sequence of executable instructions. However, current LLM-based planners are only able to operate with a fixed set of skills. We overcome this critical limitation and present a method for using LLM-based planners to query new skills and teach robots these skills in a data and time-efficient manner for rigid object manipulation. Our system can re-use newly acquired skills for future tasks, demonstrating the potential of open world and lifelong learning. We evaluate the proposed framework on multiple tasks in simulation and the real world. Videos are available at: url{https://sites.google.com/view/halp-submission}.

Keywords: Continual Learning, Probabilistic Inference, Incremental Learning
Abstract: The sense of touch is essential for robots to perform various daily tasks. Artificial Neural Networks have shown significant promise in advancing robotic tactile learning. However, due to the changing of tactile data distribution as robots encounter new tasks, ANN-based robotic tactile learning suffers from catastrophic forgetting. To solve this problem, we introduce a novel continual learning (CL) framework called the Probabilistic Spiking Neural Network with Variational Continual Learning (PSNN-VCL). In this framework, PSNN introduces uncertainty during spike emission and can apply fast Variational Inference by optimizing the uncertainty through backpropagation, which significantly reduces the required model parameters for VCL. We establish a robotic tactile CL benchmark using publicly available datasets to evaluate our method. Experimental results demonstrated that, compared to other CL methods, PSNN-VCL not only achieves superior performance in terms of widely used CL metrics but also achieves at least a 50% reduction in model parameters on the robotic tactile CL benchmark.

Keywords: Imitation Learning, Continual Learning, Deep Learning in Grasping and Manipulation
Abstract: We introduce LOTUS, a continual imitation learning algorithm that empowers a physical robot to continuously and efficiently learn to solve new manipulation tasks throughout its lifespan. The core idea behind LOTUS is constructing an ever-growing skill library from a sequence of new tasks with a small number of corresponding task demonstrations. LOTUS starts with a continual skill discovery process using an open-vocabulary vision model, which extracts skills as recurring patterns presented in unstructured demonstrations. Continual skill discovery updates existing skills to avoid catastrophic forgetting of previous tasks and adds new skills to exhibit novel behaviors. LOTUS trains a meta-controller that flexibly composes various skills to tackle vision-based manipulation tasks in the lifelong learning process. Our comprehensive experiments show that LOTUS outperforms state-of-the-art baselines by over 11% in average success rates, showing its superior knowledge transfer ability compared to prior methods. More experimental videos and results can be found on the project website: https://ut-austin-rpl.github.io/Lotus/

Keywords: Robot Safety, Reinforcement Learning, Autonomous Agents
Abstract: Safety certificates based on energy functions can provide demonstrable safety for complex robotic systems. However, all recent studies on learning-based energy function synthesis only consider the feasibility of the control policy, which might cause over-conservativeness and even fail to achieve the control goal. To solve the problem of over-conservative controllers, we proposed the magnitude regularization technique to improve the controller performance of safe controllers by reducing the conservativeness inside the energy function, while keeping the promising provable safety guarantees. Specifically, we quantify the conservativeness by the magnitude of the energy function, and we reduce the conservativeness by adding a magnitude regularization term to the synthesis loss. We propose an algorithm using reinforcement learning (RL) for synthesis to unify the learning process of safe controllers and energy functions. We conducted simulation experiments on Safety Gym and real-robot experiments using small quadrotors. Simulation results show that the proposed algorithm does reduce the conservativeness of the energy function and outperforms baselines in terms of controller performance while maintaining safety. Real-robot experiments have shown that the proposed algorithm indeed reduce conservativeness on the small quadrotors.

Keywords: Robot Safety, Reinforcement Learning, Optimization and Optimal Control
Abstract: Safety has been a critical issue for the deployment of learning-based approaches in real-world applications. To address this issue, control barrier function (CBF) and its variants have attracted extensive attention for safety-critical control. However, due to the myopic one-step nature of CBF and the lack of principled methods to design the

class-mathcal{K} functions, there are still fundamental limitations of current CBFs: optimality, stability, and feasibility. In this paper, we proposed a novel and unified approach to address these limitations with Adaptive Multi-step Control Barrier Function (AM-CBF), where we parameterize the class-K function by a neural network and train it together with the reinforcement learning policy. Moreover, to mitigate the myopic nature, we propose a novel multi-step training and single-step execution paradigm to make

CBF farsighted while the execution remains solving a single-step convex

quadratic program. Our method is evaluated on the first and second-order systems in various scenarios, where our approach outperforms the conventional CBF both qualitatively and quantitatively.

Keywords: Robot Safety, Reinforcement Learning, Robust/Adaptive Control
Abstract: Robots are more capable of achieving manipulation tasks for everyday activities than before. But the safety of manipulation skills that robots employ is still an open problem. Considering all possible failures during skill learning increases the complexity of the process and restrains learning an optimal policy. Nonetheless, safety-focused modularity in the acquisition of skills has not been adequately addressed in previous works. For that purpose, we reformulate skills as base and failure prevention skills, where base skills aim at completing tasks and failure prevention skills aim at reducing the risk of failures to occur. Then, we propose a modular and hierarchical method for safe robot manipulation by augmenting base skills by learning failure prevention skills with reinforcement learning and forming a skill library to address different safety risks. Furthermore, a skill selection policy that considers estimated risks is used for the robot to select the best control policy for safe manipulation. Our experiments show that the proposed method achieves the given goal while ensuring safety by preventing failures. We also show that with the proposed method, skill learning is feasible and our safe manipulation tools can be transferred to the real environment.

Keywords: Perception-Action Coupling, Deep Learning Methods
Abstract: A robot in a human-centric environment needs to account for the human’s intent and future motion in its task and motion planning to ensure safe and effective operation. This requires symbolic reasoning about probable future actions and the ability to tie these actions to specific locations in the physical environment. While one can train behavioral models capable of predicting human motion from past activities, this approach requires large amounts of data to achieve acceptable long-horizon predictions. More importantly, the resulting models are constrained to specific data formats and modalities. Moreover, connecting predictions from such models to the environment at hand to ensure the applicability of these predictions is an unsolved problem. We present a system that utilizes a Large Language Model (LLM) to infer a human’s next actions from a range of modalities without fine-tuning. A novel aspect of our system that is critical to robotics applications is that it links the predicted actions to specific locations in a semantic map of the environment. Our method leverages the fact that LLMs, trained on a vast corpus of text describing typical human behaviors, encode substantial world knowledge, including probable sequences of human actions and activities. We demonstrate how these localized activity predictions can be incorporated in a human-aware task planner for an assistive robot to reduce the occurrences of undesirable human-robot interactions by 29.2% on average.

Keywords: Learning from Experience, Learning from Demonstration, Sensorimotor Learning
Abstract: We developed a modality attention motion generation model on the basis of multi-modality prediction. This model provides interpretability about modality usage and demonstrates robustness against disturbances. We used a hierarchical model consisting of low-level recurrent neural networks (RNNs) for processing each modality individually and a high-level RNN that integrates the multi-modality. This integration is achieved by efficiently gating multi-modality and inputting it to the high-level RNN. We verified the interpretability and robustness of the task of inserting a furniture part, which consists of the ``approach" phase to bring the wooden dowel closer to the hole and the ``insertion" phase. While the proposed model achieves the same task success rate as the conventional model, it clarifies that it refers to vision during ``approach" and force during ``insertion," providing interpretability regarding modality use. Furthermore, in contrast to the non-modality attention model, whose task success rate drops significantly under disturbance, the proposed model enhances robustness against disturbances to modalities it does not direct attention during the task, resulting in a consistently high success rate (≒90%).

Keywords: Learning from Experience, Modeling, Control, and Learning for Soft Robots, AI-Based Methods
Abstract: Various robots have been developed so far; however, we face challenges in modeling the low-rigidity bodies of some robots. In particular, the deflection of the body changes during tool-use due to object grasping, resulting in significant shifts in the tool-tip position and the body's center of gravity. Moreover, this deflection varies depending on the weight and length of the tool, making these models exceptionally complex. However, there is currently no control or learning method that takes all of these effects into account. In this study, we propose a method for constructing a neural network that describes the mutual relationship among joint angle, visual information, and tactile information from the feet. We aim to train this network using the actual robot data and utilize it for tool-tip control. Additionally, we employ Parametric Bias to capture changes in this mutual relationship caused by variations in the weight and length of tools, enabling us to understand the characteristics of the grasped tool from the current sensor information. We apply this approach to the whole-body tool-use on KXR, a low-rigidity plastic-made humanoid robot, to validate its effectiveness.

Keywords: Learning from Experience, Probability and Statistical Methods, Transfer Learning
Abstract: Robots acting in human environments will often face new situations and can benefit from transferring prior experience. Priors could enable robots to handle new tasks zero-shot and help prevent failures, which can be particularly costly in real robot applications. Due to their interpretable nature, causal Bayesian Networks (CBN) are popular for modeling cause-effect relations between semantically meaningful environment features and their effects on action success. While the CBN structure is often intuitively transferable to a new context, its probability distribution might change, requiring data-intensive relearning. In this work, we propose three strategies that utilize semantic similarity and relatedness between the variables of two CBNs to generate and transfer informed CBN distribution priors. We evaluate the parameter prior accuracy in five different transfer scenarios, including sim-2-real, transferring parameters to more complex tasks with a larger number of parameters and even between two different tasks, which is particularly challenging. We show that the priors lead to better distribution estimates, particularly under a limited amount of new experiments, and improve the robot’s ability to predict and prevent action failures by up to 50%.

Keywords: Evolutionary Robotics, Reinforcement Learning
Abstract: Pre-training a diverse set of neural network controllers in simulation has enabled robots to adapt online to damage in robot locomotion tasks. However, finding diverse, high-performing controllers requires expensive network training and extensive tuning of a large number of hyperparameters. On the other hand, Covariance Matrix Adaptation MAP-Annealing (CMA-MAE), an evolution strategies (ES)-based quality diversity algorithm, does not have these limitations and has achieved state-of-the-art performance on standard QD benchmarks. However, CMA-MAE cannot scale to modern neural network controllers due to its quadratic complexity. We leverage efficient approximation methods in ES to propose three new CMA-MAE variants that scale to high dimensions. Our experiments show that the variants outperform ES-based baselines in benchmark robotic locomotion tasks, while being comparable with or exceeding state-of-the-art deep reinforcement learning-based quality diversity algorithms.

Keywords: Learning from Demonstration, Surgical Robotics: Laparoscopy, Learning from Experience
Abstract: In this study, we present an implementation strategy for a robot that performs peg transfer tasks in Fundamentals of Laparoscopic Surgery (FLS) via imitation learning, aimed at the development of an autonomous robot for laparoscopic surgery. Robotic laparoscopic surgery presents two main challenges: (1) the need to manipulate forceps using ports established on the body surface as fulcrums, and (2) difficulty in perceiving depth information when working with a monocular camera that displays its images on a monitor. Especially, regarding issue (2), most prior research has assumed the availability of depth images or models of a target to be operated on. Therefore, in this study, we achieve more accurate imitation learning with only monocular images by extracting motion constraints from one exemplary motion of skilled operators, collecting data based on these constraints, and conducting imitation learning based on the collected data. We implemented an overall system using two Franka Emika Panda Robot Arms and validated its effectiveness.

Keywords: Data Sets for Robot Learning, Service Robotics, Dynamics
Abstract: Most existing robotic datasets capture static scene data and thus are limited in evaluating robots' dynamic performance. To address this, we present a mobile robot oriented large-scale indoor dataset, denoted as THUD (Tsinghua University Dynamic) robotic dataset, for training and evaluating their dynamic scene understanding algorithms. Specifically, the THUD dataset construction is first detailed, including organization, acquisition, and annotation methods. It comprises both real-world and synthetic data, collected with a real robot platform and a physical simulation platform, respectively. Our current dataset includes 13 larges-scale dynamic scenarios, 90K image frames, 20M 2D/3D bounding boxes of static and dynamic objects, camera poses, and IMU. The dataset is still continuously expanding. Then, the performance of mainstream indoor scene understanding tasks, e.g. 3D object detection, semantic segmentation, and robot relocalization, is evaluated on our THUD dataset. These experiments reveal serious challenges for some robot scene understanding tasks in dynamic scenes. By sharing this dataset, we aim to foster and iterate new mobile robot algorithms quickly for robot actual working dynamic environment, i.e. complex crowded dynamic scenes.

Keywords: Data Sets for Robot Learning, Intention Recognition, Human-Robot Collaboration
Abstract: In collaborative human-robot manipulation, a robot must predict human intents and adapt its actions accordingly to smoothly execute tasks. However, the human's intent in turn depends on actions the robot takes, creating a chicken-or-egg problem. Prior methods ignore such inter-dependency and instead train marginal intent prediction models independent of robot actions. This is because training conditional models is hard given a lack of paired human-robot interaction datasets. Can we instead leverage large-scale human-human interaction data that is more easily accessible? Our key insight is to exploit a correspondence between human and robot actions that enables transfer learning from human-human to human-robot data. We propose a novel architecture, InteRACT, that pre-trains a conditional intent prediction model on large human-human datasets and fine-tunes on a small human-robot dataset. We evaluate on a set of real-world collaborative human-robot manipulation tasks and show that our conditional model improves over various marginal baselines. We also introduce new techniques to tele-operate a 7-DoF robot arm and collect a diverse range of human-robot collaborative manipulation data which we open-source. We release our code and datasets at https://portal-cornell.github.io/interact/.

Keywords: Data Sets for Robot Learning, Integrated Planning and Learning, Deep Learning Methods
Abstract: Machine Learning (ML) has replaced traditional handcrafted methods for perception and prediction in autonomous vehicles. Yet for the equally important planning task, the adoption of ML-based techniques is slow. We present nuPlan, the world’s first real-world autonomous driving dataset, and benchmark. The benchmark is designed to test the ability of ML-based planners to handle diverse driving situations and to make safe and efficient decisions. To that end, we introduce a new large-scale dataset that consists of 1282 hours of diverse driving scenarios from 4 cities (Las Vegas, Boston, Pittsburgh, and Singapore) and includes high-quality auto-labeled object tracks and traffic light data. We exhaustively mine and taxonomize common & rare driving scenarios which are used during evaluation to get fine-grained insights into the performance and characteristics of a planner. Beyond the dataset, we provide a simulation and evaluation framework that enables a planner’s actions to be simulated in closed-loop to account for interactions with other traffic participants. We present a detailed analysis of numerous baselines and investigate gaps between ML-based and traditional methods. Find the nuPlan dataset and code at nuplan.org.

Keywords: Data Sets for Robot Learning, Human-Aware Motion Planning, Data Sets for Robotic Vision
Abstract: Social navigation and pedestrian behavior research has shifted towards machine learning-based methods and converged on the topic of modeling inter-pedestrian interactions and pedestrian-robot interactions. For this, large-scale datasets that contain rich information are needed. We describe a portable data collection system, coupled with a semi-autonomous labeling pipeline. As part of the pipeline, we designed a label correction web application that facilitates human verification of automated pedestrian tracking outcomes. Our system enables large-scale data collection in diverse environments and fast trajectory label production. Compared with existing pedestrian data collection methods, our system contains three components: a combination of top-down and ego-centric views, natural human behavior in the presence of a socially appropriate "robot", and human-verified labels grounded in the metric space. To the best of our knowledge, no prior data collection system has a combination of all three components. We further introduce our ever-expanding dataset from the ongoing data collection effort -- the TBD Pedestrian Dataset and show that our collected data is larger in scale, contains richer information when compared to prior datasets with human-verified labels, and supports new research opportunities.

Keywords: Data Sets for Robot Learning, Data Sets for Robotic Vision, Learning Categories and Concepts
Abstract: We present a scalable, bottom-up and intrinsically diverse data collection scheme that can be used for high-level reasoning with long and medium horizons and that has 2.2x higher throughput compared to traditional narrow top-down step-by-step collection. We collect realistic data by performing any user requests within the entirety of 3 office buildings and using multiple embodiments (robot, human, human with grasping tool). With this data, we show that models trained on all embodiments perform better than ones trained on the robot data only, even when evaluated solely on robot episodes. We explore the economics of collection costs and find that for a fixed budget it is beneficial to take advantage of the cheaper human collection along with robot collection. We release a large and highly diverse (29,520 unique instructions) dataset dubbed RoboVQA containing 829,502 (video, text) pairs for robotics-focused visual question answering. We also demonstrate how evaluating real robot experiments with an intervention mechanism enables performing tasks to completion, making it deployable with human oversight even if imperfect while also providing a single performance metric. We demonstrate a single video-conditioned model named RoboVQA-VideoCoCa trained on our dataset that is capable of performing a variety of grounded high-level reasoning tasks in broad realistic settings with a cognitive intervention rate 46% lower than the zero-shot state of the art visual language model (VLM) baseline and is able to guide real robots through long-horizon tasks. The performance gap with zero-shot state-of-the-art models indicates that a lot of grounded data remains to be collected for real-world deployment, emphasizing the critical need for scalable data collection approaches. Finally, we show that video VLMs significantly outperform single-image VLMs with an average error rate reduction of 19% across all VQA tasks. Data and videos are available at https://robovqa.github.io

Keywords: Data Sets for Robot Learning, Big Data in Robotics and Automation, Imitation Learning
Abstract: A key challenge for robotic manipulation in open domains is how to acquire diverse and generalizable skills for robots. Recent progress in one-shot imitation learning and robotic foundation models have shown promise in transferring trained policies to new tasks based on demonstrations. This feature is attractive for enabling robots to acquire new skills and improve their manipulative ability. However, due to limitations in the training dataset, the current focus of the community has mainly been on simple cases, such as push or pick-place tasks, relying solely on visual guidance. In reality, there are many complex skills, some of which may even require both visual and tactile perception to solve. This paper aims to unlock the potential for an agent to generalize to hundreds of real-world skills with multi-modal perception. To achieve this, we have collected a dataset comprising over 110,000 contact-rich robot manipulation sequences across diverse skills, contexts, robots, and camera viewpoints, all collected in the real world. Each sequence in the dataset includes visual, force, audio, and action information. Moreover, we also provide a corresponding human demonstration video and a language description for each robot sequence. We have invested significant efforts in calibrating all the sensors and ensuring a high-quality dataset. The dataset is made publicly available on our website: https://rh20t.github.io.

Keywords: Data Sets for Robot Learning, Machine Learning for Robot Control
Abstract: Machine learning provides a powerful tool for building socially compliant robotic systems that go beyond simple predictive models of human behavior. By observing and understanding human interactions from past experiences, learning can enable effective social navigation behaviors directly from data. In this paper, our goal is to develop methods for training policies for socially unobtrusive behavior, such that robots can navigate among humans in ways that don't disturb human behavior in visual navigation using only onboard RGB observations. We introduce a definition for such behavior based on the counterfactual perturbation of the human: if the robot had not intruded into the space, would the human have acted in the same way? By minimizing this counterfactual perturbation, we can induce robots to behave in ways that do not alter the natural behavior of humans in the shared space. Instantiating this principle requires training policies to minimize their effect on human behavior, and this in turn requires data that allows us to model the behavior of humans in the presence of robots. Therefore, our approach is based on two key contributions. First, we collect a large dataset where an indoor mobile robot interacts with human bystanders. Second, we utilize this dataset to train policies that minimize counterfactual perturbation. We provide supplementary videos and make publicly available the visual navigation dataset on our project page.

Keywords: Soft Robot Materials and Design, Biologically-Inspired Robots
Abstract: Aquatic organisms, due to soft body structure and high agility, have inspired many biomimetic robots. However, considering the issues of insulation and waterproofing, as well as the driving module of soft materials, their control systems are usually larger and heavier. Therefore, small underwater robots often tethered, i.e., it cannot integrate energy and control systems onto the body, which greatly limited in its working range and activity mode. This paper presents a small untethered bionic manta ray. The robotic fish is driven by dielectric elastomer actuators (DEA), which controls the double wing structure on both sides by the central muscle part to simulate the process of the manta ray’s lateral fins fanning to propel itself forward. And the flexible printed circuit board (FPC) constitutes the body of the fish and is also an independent energy control system. The electronic components are evenly distributed on the double-wing structure of the robotic fish to realize the integration of the energy control system. This circuit system can be powered by a small lithium battery and output a periodic voltage to drive the motion of the robotic fish. The masses of our tethered and untethered fish are 1.9g and 5.1g respectively. The swimming speed of these two types of fish can reach 42.5mm/s and 17.0 mm/s. And this design principle can be extended to the research and design of various flexible devices and soft robots.

Keywords: Soft Robot Materials and Design, Compliant Joints and Mechanisms, Optimization and Optimal Control
Abstract: Robot mechanisms that exploit compliance can perform complex tasks under uncertainty using simple control strategies, but it remains difficult to design mechanisms with a desired embodied intelligence. In this article, we propose an automated design technique that optimizes the desired load-displacement behavior of planar flexible-link mechanisms. To do so, we replace a subset of rigid with flexible links in an existing mechanism, and optimize their rest shape. We demonstrate the efficacy of our approach on a set of examples, including two fabricated prototypes, illustrating applications for grasping and locomotion tasks.

Keywords: Soft Robot Materials and Design, Flexible Robotics
Abstract: Research in the area of photo-actuation is growing rapidly, yet there are few examples of photo-actuators with practical use cases. One potential application is for the control of intelligent electromagnetic surfaces, or two-dimensional ar- rays that could shape and control an incident electromagnetic field in ideally any manner. A promising concept to realize such a surface leverages signal refraction via antenna edges, but requires non-metal actuation, large antenna rotations, and high antenna angular accuracy for long periods of time. Here, we present a nonmetal, light-controlled, multi-position inchworm actuator array that can rotate an antenna 88 degrees in incre- mental steps of less than 3.4 degrees with zero-power shape- persistence. The design is modular and rapidly manufacturable via a layered laser-cutting technique, such that the actuator can be tiled into an array to control the rotation of many antennas. We control the array with a single focused IR light that rasters across the actuators to precisely control all antenna positions. We characterize the response time, accuracy, and repeatability of a single actuator, and demonstrate the array achieving diverse antenna configurations. This work advances the precision and scalability of photothermal actuation not only for use in intelligent electromagnetic surfaces but for any application benefitting from light-controlled actuation.

Keywords: Soft Robot Materials and Design, Force and Tactile Sensing, Mechanism Design
Abstract: Many robotic hands currently rely on extremely dexterous robotic fingers and a thumb joint to envelop themselves around an object. Few hands focus on the palm even though human hands greatly benefit from their central fold and soft surface. As such, we develop a novel structurally compliant soft palm, which enables more surface area contact for the objects that are pressed into it. Moreover, this design, along with the development of a new low-cost, flexible illumination system, is able to incorporate a high-resolution tactile sensing system inspired by the GelSight sensors. Concurrently, we design RObotic Modular Endoskeleton Optical (ROMEO) fingers, which are underactuated two-segment soft fingers that are able to house the new illumination system, and we integrate them into these various palm configurations. The resulting robotic hand is slightly bigger than a baseball and represents one of the first soft robotic hands with actuated fingers and a passively compliant palm, all of which have high-resolution tactile sensing. This design also potentially helps researchers discover and explore more soft-rigid tactile robotic hand designs with greater capabilities in the future.

Keywords: Soft Robot Materials and Design, Grippers and Other End-Effectors, Biomimetics
Abstract: Suction is widely used in industry, but the adaptation of state-of-the-art suction cups on complex surfaces (i.e., curved, cornered, uneven, rough, etc.) are still limited. In this paper, we present a novel shape-conformable suction mechanism to achieve highly-adaptive suction on complex surfaces. The shape-conformable adaptive suction is obtained by squeezing a soft multi-layer structure on the substrate, to form a shape-to-roughness sealed suction region. Based on this mechanism, two shape-conformable suction cups (SCSCs) - a displacement-driven shape-conformable suction cup (SDisp) and a force-driven shape-conformable suction cup (SForce) - are designed. They both achieve highly-adaptive suction on challenging surface topographies including highly-curved, cornered, textured, uneven and tilted surfaces. Particularly, SDisp has better adaptation (e.g., on a 90-degree corner and a balloon) and SForce is more lightweight (26 g) and compact (46×35 mm), and exhibits quicker suction response (0.4 s). We analyse the underlying adaptive suction mechanism by the physical model, and demonstrate its adaptive suction capability by qualitatively comparing it with previous suction cups. We finally conclude design principles for improving suction adaptation. We believe the proposed shape-conformable suction mechanism provides a novel solution to realize adaptive suction on complex surfaces in next-generation robotic gripping, anchoring and manipulation.

Keywords: Soft Robot Materials and Design, Grippers and Other End-Effectors, Soft Robot Applications
Abstract: The inherent elasticity of soft materials can be used to create robotic grippers that deform and comply to a variety of irregular shapes. To date, several soft adaptive grasping strategies have been reported, however, most of them focus on adapting to the overall shape of the structure, while the adaptive grasping of small surface asperities is overlooked. In this paper, we propose a novel method to achieve adaptive grasping on surface asperities with a smart shape-memory silicone sponge. Heating above 60◦C makes the sponge soft and deformable to allow it to penetrate within surface asperities via a pressure normal to the surface. Cooling down below 60◦C makes the sponge “jam” to retain its deformed shape. The interlocking force between the jammed sponge and the asperities, and the increased area of contact, allows for adaptive grasping on asperities down to 0.4 mm with an adhesive force of up to 27.7 N in a 40 × 40 mm contacting area. We introduce the design, working principle, fabrication, and optimization of a robotic gripper based on this shape-memory silicone sponge. This sponge-jamming gripper shows great potential for developing next-generation robotic grippers for the manipulation of textured and discontinuous surfaces.

Keywords: Soft Robot Materials and Design, Mechanism Design, Additive Manufacturing
Abstract: Origami designs and structures have been widely used in many fields, such as morphing structures, robotics, and metamaterials. However, the design and fabrication of origami structures rely on human experiences and skills, which are both time and labor-consuming. In this paper, we present a rapid design and fabrication method for string-driven origami structures and robots. We developed an origami design software to generate desired crease patterns based on analytical models and Evolution Strategies (ES). Additionally, the software can automatically produce 3D models of origami designs. We then used a dual-material 3D printer to fabricate those wrapping-based origami structures with the required mechanical properties. We utilized Twisted String Actuators (TSAs) to fold the target 3D structures from flat plates. To demonstrate the capability of these techniques, we built and tested an origami crawling robot and an origami robotic arm using 3D-printed origami structures driven by TSAs.

Keywords: Soft Robot Materials and Design, Micro/Nano Robots, Soft Robot Applications
Abstract: Magnetic manipulation of miniature soft or liquid robots capable of deformation has gained increasing attention and is demonstrating great potential in small-scale applications, such as drug delivery, minimal invasive surgery, and manipulation of delicate objects. In this study, we introduce a liquid robot composed of ferrofluid that shows promise for small-scale magnetic manipulation applications. The objective of this work is to achieve more flexible manipulation capabilities of the robot. To this end, we utilize a redundant magnetic actuation system composed of five electromagnets and implement 4 degrees of freedom (4-DOF) control of the liquid robot in planar space. Based on the planar 4-DOF control, the liquid robot is able to perform various actions and implement versatile manipulation tasks, such as transporting objects, separating or assembling miniature parts, and operating customized tools. Furthermore, we suggest an automatic transportation method to enhance manipulation precision. A series of experiments are conducted to validate the effectiveness of the proposed method and the robot's capacity to accomplish diversified manipulation tasks. The proposed liquid robot indicates flexibility and provides novel solutions for small-scale untethered manipulation.

Keywords: Soft Robot Materials and Design, Modeling, Control, and Learning for Soft Robots, Optimization and Optimal Control
Abstract: Dielectric elastomer actuators (DEAs), a type of "artificial muscles", can generate significant deformations and offer speedy responses when exposed to voltage. Owing to their high electromechanical conversion efficiency and great flexibility, they have been extensively used in soft robot applications, such as soft grippers, walking robots, crawling robots, climbing robots, swimming robots, etc. Although previous research has explored the use of DEAs in soft robot locomotion, achieving optimal behavior is challenging due to the complexity of the constituent materials and the highly nonlinear nature of the problem. In this study, a simulation-based design optimization approach is proposed to address this challenge. The proposed approach involves developing a computational modeling framework that evaluates the electromechanical behavior of the DEA. A graph neural network (GNN) is employed as an encoder to extract the latent representation of the geometry in a low dimensional space, which is further used to construct a surrogate model for fast prediction of target responses. To achieve an optimal actuation capability under design constraints, a multi-objective optimization function is formulated to balance the actuation distance and the actuator size, where the Pareto front demonstrates the trade-off between the actuation distance and design constraint. Finally, three optimized designs are fabricated and tested, demonstrating a performance improvement of over 140% compared to an

Keywords: Deep Learning for Visual Perception, Computer Vision for Automation, RGB-D Perception
Abstract: Point cloud primitive segmentation aims to segment the surface point cloud into various geometric types of primitives, which plays a vital role in robot operation and industrial automation. However, differences in object structures and shapes across industrial datasets create domain shift issues, compounded by privacy concerns preventing dataset sharing. To address these challenges, we propose a novel source-free domain adaptation method for point cloud primitive segmentation, which follows the popular pseudo-label based self-training framework. Unlike previous works using single-model uncertainty to refine pseudo labels, our method leverages multi-confidence, including transformation consistency, task confidence, and geometric saliency to provide more informative guidance. Specifically, the transformation consistency is first utilized to vote pseudo-labels and task confidences. Furthermore, to filter out high-confident noises and obtain more reliable pseudo-labels, we investigate the geometric curvature properties of primitives and propose a geometric saliency guided dynamic prototype matching and label graph aggregation strategies for pseudo-label reassignment with different task confidence. For this novel task, we construct several datasets and verify the effectiveness of the proposed methods through a series of experiments.

Keywords: Deep Learning for Visual Perception, Deep Learning Methods, Localization
Abstract: Cross-modality point cloud registration is confronted with significant challenges due to inherent differences in modalities between sensors. To deal with this problem, we propose FF-LOGO: a cross-modality point cloud registration framework with Feature Filtering and LOcal-Global Optimization. The cross-modality feature correlation filtering module extracts geometric transformation-invariant features from cross-modality point clouds and achieves point selection by feature matching. We also introduce a cross-modality optimization process, including a local adaptive key region aggregation module and a global modality consistency fusion optimization module. Experimental results demonstrate that our two-stage optimization significantly improves the registration accuracy of the feature association and selection module. Our method achieves a substantial increase in recall rate compared to the current state-of-the-art methods on the 3DCSR dataset, improving from 40.59% to 75.74%. Our code will be available at https://github.com/wangmohan17/FFLOGO.

Keywords: Deep Learning for Visual Perception, RGB-D Perception, Recognition
Abstract: The ability to estimate joint parameters is essential for various applications in robotics and computer vision. In this paper, we propose CAPT: category-level articulation estimation from a point cloud using Transformer. CAPT uses an end-to-end transformer-based architecture for joint parameter and state estimation of articulated objects from a single point cloud. The proposed CAPT methods accurately estimate joint parameters and states for various articulated objects with high precision and robustness. The paper also introduces a motion loss approach, which improves articulation estimation performance by emphasizing the dynamic features of articulated objects. Additionally, the paper presents a double voting strategy to provide the framework with coarse-to-fine parameter estimation. Experimental results on several category datasets demonstrate that our methods outperform existing alternatives for articulation estimation. Our research provides a promising solution for applying Transformer-based architectures in articulated object analysis.

Keywords: Deep Learning for Visual Perception, Semantic Scene Understanding, Recognition
Abstract: Autonomous vehicles rely on LiDAR sensors to perceive the environment. Adverse weather conditions like rain, snow, and fog negatively affect these sensors, reducing their reliability by introducing unwanted noise in the measurements. In this work, we tackle this problem by proposing a novel approach for detecting adverse weather effects in LiDAR data. We reformulate this problem as an outlier detection task and use an energy-based framework to detect outliers in point clouds. More specifically, our method learns to associate low energy scores with inlier points and high energy scores with outliers allowing for robust detection of adverse weather effects. In extensive experiments, we show that our method performs better in adverse weather detection and has higher robustness to unseen weather effects than previous state-of-the-art methods. Furthermore, we show how our method can be used to perform simultaneous outlier detection and semantic segmentation. Finally, to help expand the research field of LiDAR perception in adverse weather, we release the SemanticSpray dataset, which contains labeled vehicle spray data in highway-like scenarios. The dataset will be available upon acceptance (see supplementary material for sample).

Keywords: Deep Learning for Visual Perception, Vision-Based Navigation
Abstract: Efficient interest point detection and description in images play a crucial role in many tasks such as multi-robot SLAM and collaborative localization. To facilitate fast detection and generate compact descriptions on edge devices, we introduce EdgePoint, a lightweight neural network. We design a new detection loss UnfoldSoftmax to improve inference speed. Futhermore, we propose Ortho-Alignment loss combined with LocalPCA compression to learn compact 32-dimensional descriptors. To enable efficient storage or communication, we also quantize the generated descriptors into integral values. We perform EdgePoint on various datasets, and show that it surpasses SuperPoint in performance while utilizing only 1% of the parameters and achieving up to more than 10 times faster inference speed. By applying descriptor quantization, the requirements for storage and communication can be reduced by up to 97% without performance decreasing.

Keywords: Deep Learning for Visual Perception, Visual Learning
Abstract: Point cloud registration is essential in computer vision and robotics. Recently, transformer-based methods have achieved advanced point cloud registration performance. However, the standard attention mechanism utilized in these methods considers many low-relevance points, and it has difficulty focusing its attention weights on sparse and meaningful points, leading to limited local structure modeling capabilities and quadratic computational complexity. To address these limitations, we present the Tree-based Transformer (TrT), which is able to extract abundant local and global features with linear computational complexity. Specifically, the TrT builds coarse-to-dense feature trees, and a novel Tree-based Attention (TrA) is proposed to guide the progressive convergence of the attended regions toward meaningful points and to structurize point clouds following tree structures. In each layer, the top S key points with the highest attention scores are selected, such that in the next layer, attention is evaluated only within the specified high-relevance regions, corresponding to the child points of these selected S points. Additionally, coarse features containing high-level semantic information are incorporated into the child points to guide the feature extraction process, facilitating local structure modeling and multiscale information integration. Consequently, TrA enables the model to focus on critical local structures and extract rich local information with linear computational complexity. Experiments demonstrate that our method achieves state-of-the-art performance on 3DMatch and KITTI benchmarks. The code for our method is publicly available at https://github.com/CGuangyan-BIT/TrT.

Keywords: Deep Learning in Grasping and Manipulation
Abstract: Existing grasp prediction approaches are mostly based on offline learning, while, ignoring the exploratory grasp learning during online adaptation to new picking scenarios, i.e., objects that are unseen or out-of-domain (OOD), camera and bin settings, etc. In this paper, we present an uncertainty-based approach for online learning of grasp predictions for robotic bin picking. Specifically, the online learning algorithm with an effective exploration strategy can significantly improve its adaptation performance to unseen environment settings. To this end, we first propose to formulate online grasp learning as an RL problem that will allow us to adapt both grasp reward prediction and grasp poses. We propose various uncertainty estimation schemes based on emph{Bayesian uncertainty quantification} and emph{distributional ensembles}. We carry out evaluations on real-world bin picking scenes of varying difficulty. The objects in the bin have various challenging physical and perceptual characteristics that can be characterized by semi- or total transparency, and irregular or curved surfaces. The results of our experiments demonstrate a notable improvement of grasp performance in comparison to conventional online learning methods which incorporate only naive exploration strategies. Video: https://youtu.be/fPKOrjC2QrU

Keywords: Deep Learning in Grasping and Manipulation
Abstract: The prevailing grasp prediction methods predominantly rely on offline learning, overlooking the dynamic grasp learning that occurs during real-time adaptation to novel picking scenarios. These scenarios may involve previously unseen objects, variations in camera perspectives, and bin configurations, among other factors. In this paper, we introduce introduce a novel approach, SSL-ConvSAC, that combines semi-supervised learning and reinforcement learning for online grasp learning. By treating pixels with reward feedback as labeled data and others as unlabeled, it efficiently exploits unlabeled data to enhance learning. In addition, we address the imbalance between labeled and unlabeled data by proposing a contextual curriculum-based method. We ablate the proposed approach on real-world evaluation data and demonstrate promise for improving online grasp learning on bin picking tasks using a physical 7-DoF Franka Emika robot arm with a suction gripper.

Keywords: Deep Learning in Grasping and Manipulation
Abstract: Visual-based robot pose estimation is a fundamen- tal challenge, involving the determination of the camera’s pose with respect to a robot. Conventional methods for camera-to- robot pose calibration rely on fiducial markers to establish keypoint correspondences. However, these approaches exhibit significant variability in accuracy and robustness, particularly in 2D keypoint detection. In this work, we present an end- to-end pose estimation approach that achieves camera-to-robot calibration using monocular images and keypoint information. Our method employs a two-level nested U-shaped architecture, featuring a bottom-level residual U-block to extract richer contextual information from diverse receptive fields to enhance keypoint refinement. By incorporating the perspective-n-point (PnP) algorithm and leveraging 3D robot joint keypoints, we establish correspondence of 3D coordinate points between the robot’s coordinate system and the camera’s coordinate system, facilitating accurate pose estimation. Experimental evaluations encompass real-world and synthetic datasets, demonstrating competitive results across three distinct robot manipulators.

Keywords: Deep Learning in Grasping and Manipulation, Deep Learning Methods, Failure Detection and Recovery
Abstract: This paper introduces a method, Generative Adversarial Networks for Detecting Erroneous Results (GANDER), leveraging Generative Adversarial Networks to provide online error detection in manipulation tasks for autonomous robot systems. GANDER relies on mapping input images of a trained task to a learned manifold that contains only positive task executions and outcomes. When reconstructed through this manifold, the input images from successful task executions will remain largely unchanged, while the images from a failed task will change significantly. Using this insight, GANDER enables inspection and task outcome verification capabilities using a large number of positive examples but only a small set of negative examples, thus increasing the applicability of autonomous robot systems. We detail the design of GANDER and provide results of a proof-of-concept system, establishing its efficacy in an autonomous inspection, maintenance, and repair task. GANDER produces favorable results compared to baseline approaches and is capable of correctly identifying off-nominal behavior with 91.65% accuracy in our test task. Ablation studies were also performed to quantify the amount of data ultimately needed for this approach to succeed.

Keywords: Deep Learning in Grasping and Manipulation, Computer Vision for Automation, Recognition
Abstract: Task-oriented grasping models aim to predict a suitable grasp pose on an object to fulfill a task. These systems have limited generalization capabilities to new tasks, but have shown the ability to generalize to novel objects by recognizing affordances. This object generalization comes at the cost of being unable to recognize the object category being grasped, which could lead to unpredictable or risky behaviors. To overcome these generalization limitations, we contribute a novel system for task-oriented grasping called the One-shot Task-oriented Grasping (OS-TOG) framework. OS-TOG comprises four interchangeable neural networks that interact through dependable reasoning components, resulting in a single system that predicts multiple grasp candidates for a specific object and task from multi-object scenes. Embedded one-shot learning models leverage references within a database for OS-TOG to generalize to novel objects and tasks more efficiently than existing alternatives. Additionally, the paper presents suitable candidates for the framework’s neural components, covering essential adjustments for their integration and evaluative comparisons to state-of-the-art. In physical experiments with novel objects, OS-TOG recognizes 69.4% of detected objects correctly and predicts suitable task-oriented grasps with 82.3% accuracy, having a physical grasp success rate of 82.3%.

Keywords: Deep Learning in Grasping and Manipulation, Assembly, Big Data in Robotics and Automation
Abstract: Automating the assembly of objects from their parts is a complex problem with innumerable applications in manufacturing, maintenance, and recycling. Unlike existing research, which is limited to target segmentation, pose regression, or using fixed target blueprints, our work presents a holistic multi-level framework for part assembly planning consisting of part assembly sequence inference, part motion planning, and robot contact optimization. We present the Part Assembly Sequence Transformer (PAST) -- a sequence-to-sequence neural network -- to infer assembly sequences recursively from a target blueprint. We then use a motion planner and optimization to generate part movements and contacts. To train PAST, we introduce D4PAS: a large-scale Dataset for Part Assembly Sequences consisting of physically valid sequences for industrial objects. Experimental results show that our approach generalizes better than prior methods while needing significantly less computational time for inference.

Keywords: Deep Learning in Grasping and Manipulation
Abstract: Advancements in robot learning for object manipulation have shown promising results, yet certain everyday objects remain challenging for robots to effectively interact with. This discrepancy arises from the fact that human-designed objects are optimized for human use rather than robot manipulation. To address this gap, we propose a framework to automatically design 3D printable adaptations that can be attached to hard-to-use objects, thus improving "robot ergonomics". Our learning-based framework formulates the adaptation design and control as a dual Markov decision process and is able to improve robot-object interactions for various robot end effectors and objects. We further validate our designs in the real world with a Franka Panda robot. Please see the supplementary video and https://object-adaptation.github.io for additional visualizations.

Keywords: Deep Learning in Grasping and Manipulation, Dual Arm Manipulation, Transfer Learning
Abstract: This paper focuses on benchmarking the robustness of visual representations toward policy learning in the context of object assembly tasks, particularly the alignment and insertion of objects with geometrical extrusions, commonly known as a peg-in-hole task. The accuracy required to detect and orient the peg and the hole geometry in SE(3) space for successful assembly poses significant challenges. Addressing this, we employ a general framework in visuomotor policy learning that utilizes visual pretraining models as vision encoders. Our study investigates the robustness of this framework when applied to a dual-arm manipulation setup, specifically to the grasp variations. Our quantitative analysis shows that existing pretrained models fail to capture the essential visual features necessary for this task. However, a visual encoder trained from scratch consistently outperforms the frozen pretrained models. Moreover, we discuss rotation representations and associated loss functions that substantially improve policy learning. We present a novel task scenario designed to evaluate the progress in visuomotor policy learning, with a specific focus on improving the robustness of intricate assembly tasks that require both geometrical and spatial reasoning. Videos, data, and code for our simulator and evaluation are available at https://sites.google.com/view/geometric-peg-in-hole.

Keywords: Deep Learning in Grasping and Manipulation, Cooperating Robots, Learning from Experience
Abstract: Recent research efforts have yielded significant advancements in manipulating objects under homogeneous settings where the robot is required to either manipulate rigid or deformable (soft) objects. However, the manipulation under heterogeneous setups that involve both 1D deformable and rigid objects remains an unexplored area of research. Such setups are common in various scenarios that involve the transportation of heavy objects via ropes, e.g., on factory floors, at disaster sites, and in forestry. To address this challenge, we introduce DeRi-Bot, the first framework that enables the collaborative manipulation of rigid objects with deformable objects. Our framework comprises an Action Prediction Network (APN) and a Configuration Prediction Network (CPN) to model the complex pattern and stochasticity

of soft-rigid body systems. We demonstrate the effectiveness of DeRi-Bot in moving rigid objects to a target position with ropes connected to robotic arms. Furthermore, DeRi-Bot is a distributive method that can accommodate an arbitrary number of robots or human partners without reconfiguration or retraining. We evaluate our framework in both simulated and real-world environments and show that it achieves promising results with strong generalization across different types of objects and multi-agent settings, including human-robot collaboration.

Keywords: Physical Human-Robot Interaction, Compliance and Impedance Control, Human-Robot Collaboration
Abstract: When interacting with unmodelled dynamic systems, a robot controller should be capable of adapting online its behavior, in order to be robust to the changing environmental conditions. In the paradigm of passivity-based control, virtual energy tanks allow to perform such adaptations in a robustly stable way, by bounding the amount of energy allocated to the controller. Nevertheless, when the workspace is shared

with human collaborators, additional limits have to be imposed to the power the system can exert, in order to guarantee the overall safety. These bounds are difficult to estimate a priori, might vary over time and can significantly affect task execution. In this letter, we tackle this problem by simultaneously adapting online the admittance and the power limits in the controller, ensuring both safety and task performance. Experimental results with a collaborative manipulator validate the presented framework.

Keywords: Physical Human-Robot Interaction, AI-Based Methods, Human-Centered Robotics
Abstract: Advances in human sensing and machine learning are paving the way for new applications of robotics in sports and fitness, making skill coaching smarter, easier and more accessible. Physical and social human robot interaction in particular has received special attention as a feedback mechanism for human performance augmentation. A core challenge in deploying robots that interact physically with humans in dynamic environments such as sports, relates to modeling human skills and designing appropriate interaction schemes. We present the first ML-based HRI platform for physical robot to human skill coaching in real-time in Martial Arts which can be extended to various sports. Our system comprises of the Sawyer robot, our specially developed IoT katana and a skill-training program for the Martial Art of Iaido. We built and deployed in real-time a ML-based Iaido strike recognition model trained on expert and beginner data, and achieved accuracies ranging between 94.8% and 99.97%. We assessed the system's effectiveness in coaching skills through robot interaction in a sparring experiment and a survey involving 12 participants practicing key Iaido techniques with guided training from Sawyer. Our results demonstrated improvement in all participants' Iaido strike skill after training with Sawyer, and they responded positively to robot-assisted skill coaching.

Keywords: Physical Human-Robot Interaction, Compliance and Impedance Control, Learning from Demonstration
Abstract: Traditional strength and conditioning training relies on the utilization of free weights, such as weighted implements, to elicit external stimuli. However, this approach poses a significant challenge when attempting to modify or adjust the loads within a single training set. This paper introduces an innovative method for achieving adjustable loads during resistance training by leveraging physical Human-Robot Interaction (pHRI). The primary objective is to regulate targeted muscle activation through the use of Robo-Coach (robotic coach system). We first utilize a Task-Parameterized Gaussian Mixture Model (TP-GMM) to learn the motion of coach demonstration, which can be generalized for the trainees. The 3D path extracted from the generated trajectory is then projected onto a 2D plane with respect to the direction of the load. Furthermore, we propose a hybrid stiffness/position generator for online task execution. This generator determines the desired positions in the 2D plane according to the contact point displacements in the stimuli direction and, simultaneously, sets the desired stiffness based on the muscle activation feedback. Finally, the Robo-Coach is implemented with a variable impedance controller to achieve load-adjustable resistance training with the trainee. The biceps curl exercises were conducted and the results showed favorable performance, indicating the effectiveness of this approach.

Keywords: Physical Human-Robot Interaction, Human-Aware Motion Planning, Optimization and Optimal Control
Abstract: This paper presents a novel concept to support physically impaired humans in daily object manipulation tasks with a robot. Given a user’s manipulation sequence, we propose a predictive model that uniquely casts the user’s sequential behavior as well as a robot support intervention into a hierarchical multi-objective optimization problem. A major contribution is the prediction formulation, which allows to consider several different future paths concurrently. The second contribution is the encoding of a general notion of constancy constraints, which allows to consider dependencies between consecutive or far apart keyframes (in time or space) of a sequential task. We perform numerical studies, simulations and robot experiments to analyse and evaluate the proposed method in several table top tasks where a robot supports impaired users by predicting their posture and proactively re-arranging objects.

Keywords: Physical Human-Robot Interaction, Compliance and Impedance Control, Robust/Adaptive Control
Abstract: Haptic upper limb exoskeletons are robots that assist human operators during task execution while having the ability to render virtual or remote environments. Therefore, ensuring the stability of such robots in physical human-robot-environment interaction (pHREI) is crucial. Having a wide range of Z-width, which indicates the region of passively renderable impedance by a haptic display, is also important for rendering a broad range of virtual environments. To address these issues, this study designs subsystem-based adaptive impedance control to achieve a stable pHREI for a 7 degrees of freedom haptic exoskeleton. The presented controller decomposes the entire system into subsystems and designs controller at the subsystem level. The stability of controller in the presence of contact with the virtual environment and human arm force is proven by employing the concept of virtual stability. Additionally, the Z-width of 7-DoF haptic exoskeleton is illustrated using experimental data and improved by exploiting varying virtual mass element. Finally, experimental results are provided to demonstrate the performance of the proposed controller. The control results are also compared to state-of-the-art control methods, highlighting the excellence of the designed controller.

Keywords: Physical Human-Robot Interaction, Bimanual Manipulation, Human-Centered Robotics
Abstract: Robots have a great potential to help people with movement limitations in activities of daily living, such as dressing. A common problem in almost all dressing tasks is the insertion of a garment’s opening around a part of the human body. The rich contact environment and the deformations of the garment make the task a challenging problem for robots. In this paper, we propose a bi-manual control method for garment opening insertion during robot-assisted dressing. Specifically, we propose a model predictive controller that uses an Attention-based Relational Graph Convolutional Network (ARGCN) for modeling the dynamics of the opening in the presence of the body. We train the model entirely in simulation and validate our method in four real-world dressing scenarios of a medical training manikin. We show that our method generalizes well in the real-world opening insertion tasks achieving an overall success rate of 97.5%, even though the dynamics and the shapes vastly differ from the simulation setup.

Keywords: Physical Human-Robot Interaction, Human Performance Augmentation, Virtual Reality and Interfaces
Abstract: Exergames have been considered an advanced approach for enhancing the physical activities of the elderly community. Advancement includes greater immersive and enjoyment factors, which compute performance validation and sustain engagement to activate them from sedentary lifestyles. As a result, an at-home-based game design combined with squat workouts is essential to enhance lower-limb performance. We designed a virtual reality (VR) based exergame with dynamic difficulty adjustment (DDA) conditions in which the speed of the moving objects and air pressure of the PGM vary with respect to the knee shakiness feature. This letter aims to estimate the muscle loading and unloading effects of exergaming sessions at the onset and during the squat phase of the squat cycle. We acquired surface electromyography (sEMG) of five major lower limb muscles for seven subjects to evaluate the significant reduction in muscle activity during conventional and exergame-based squat sessions. In addition, we assessed the knee indicators to identify the variation in motion performance. The subjects performed 120 squats per session, followed by a maximum voluntary contraction (MVC) task. No adverse events, such as fatigue and dizziness, were reported during the study. Our results show a higher significant p<0.01 muscle activity difference for all tested muscles. We also found that knee shakiness showed a statistically significant reduction of p<0.01 during exergaming sessions (2 and 3).

Keywords: Physical Human-Robot Interaction, Human-Centered Robotics, Rehabilitation Robotics
Abstract: The emergence of exoskeleton technology has enabled new opportunities for gait rehabilitation, but effective methods to restore healthy gait patterns with exoskeletons are not yet clearly understood. Early research in robot-based gait rehabilitation offered little improvement over current standards of physical care and emphasized the need for a deeper understanding of the complex interaction between humans and the robot, i.e., physical human-robot interaction (pHRI). Studies reported varied lower limb responses for a similar intervention with different exoskeletons, implying that the exoskeleton's external dynamics affect musculoskeletal adaptation outcomes. Accordingly, the current study aims at showcasing the external dynamics dependent gait adaptation using a Cable-Driven Leg Exoskeleton (CDLE). A swing phase gait intervention using three different CDLE cable-routing configurations that impose varied dynamics at human anatomical joints is studied. Twenty-four healthy participants, eight for each CDLE configuration, were tested. Results showed varied gait adaptation among the three groups such that the subjects used either predominantly their hip joint, knee joint, or a combination of both joints implying selective joint strategy adaptations for different external dynamics conditions for the same intervention. The results of this study can provide insights into the optimal design of leg exoskeleton-based rehabilitation paradigms for effective gait rehabilitation.

Keywords: Physical Human-Robot Interaction, Human-Robot Collaboration, Human Factors and Human-in-the-Loop
Abstract: The Rating Scale method has been long deemed the standard for measuring subjective perceptions. However, in the field of physical human-robot collaboration (pHRC), its aptness should be put under scrutiny due to inherent challenges such as response bias, between-subject variations, and the granularity nature.

Individual variances can introduce significant bias in the rating scale results. A high granularity in the scale could overwhelm participants, leading to unclear and biased responses, while a low granularity may gloss over the fine nuances of human feelings. Additionally, there’s a notable risk of receiving careless responses, which compromise data reliability. Recognizing these challenges, this paper proposes the application of Pairwise Comparison (PC) in pHRC — an alternative survey technique that emphasizes direct comparisons between items on the defined criteria. By using the NASA Task Load Index (NASA-TLX) as a template, RS and PC questionnaires are designed and used in a series of pHRC experiments. Our preliminary findings suggest that PC is more precise and robust than the rating scale method. Compared to RS, PC fosters authentic participant interests in the experiment by intuitive question design and reducing the experimental duration. Besides, the accuracy and reliability of PC are also found to be consistent regardless of the variations in our experimental procedure design.

Keywords: Prosthetics and Exoskeletons, Actuation and Joint Mechanisms, Mechanism Design
Abstract: Prosthetic legs have been used to restore function in the lower limbs lost due to amputation. Early designs including prosthetic legs with a passive joint or without any joint as well as the Energy Storing and Releasing (ESR) feet have shown deficiency in push-off torque, which results in asymmetric gait pattern, slower walking speed, and higher cost of transportation. Although powered prosthetic legs address the aforementioned problems, they suffer from lower energy efficiency, higher volume and weight. In this paper, a powered transtibial prosthesis using a Parallel Elastic Actuator (PEA) is proposed in order to generate the joint torque needed for walking with a lower-powered actuator for lighter and more compact design. A non-linear spring mechanism is proposed to generate the spring torque as needed. The implemented prosthetic leg is evaluated with three intact subjects. The experimental results shows that smaller torque is required for the motor with the spring mechanism. Therefore, less electrical power is consumed when the spring mechanism is used, which implies a lower-powered actuator is sufficient to generate the joint torque needed for walking.

Keywords: Prosthetics and Exoskeletons, Computer Vision for Medical Robotics, Visual Tracking
Abstract: In Total Knee Replacement Arthroplasty (TKA), surgical robotics can provide image-guided navigation to fit implants with high precision. Its tracking approach highly relies on inserting bone pins into the bones tracked by the optical tracking system. This is normally done by invasive, radiative manners (implantable markers and CT scans), which introduce unnecessary trauma and prolong the preparation time for patients. To tackle this issue, ultrasound-based bone tracking could offer an alternative. In this study, we proposed a novel deep-learning structure to improve the accuracy of bone tracking by an A-mode ultrasound (US). We first obtained a set of ultrasound dataset from the cadaver experiment, where the ground truth locations of bones were calculated using bone pins. These data were used to train the proposed CasAtt-UNet to predict bone location automatically and robustly. The ground truth bone locations and those locations of US were recorded simultaneously. Therefore, we could label bone peaks in the raw US signals. As a result, our method achieved sub-millimeter precision across all eight bone areas with the only exception of one channel in the ankle. This method enables the robust measurement of lower extremity bone positions from 1D raw ultrasound signals. It shows great potential to apply A-mode ultrasound in orthopedic surgery from safe, convenient, and efficient perspectives.

Keywords: Prosthetics and Exoskeletons, Deep Learning Methods, Physical Human-Robot Interaction
Abstract: In the field of human-robot interaction, surface electromyography (sEMG) provides a valuable tool for measuring active muscular effort. While numerous studies have investigated real-time control of upper extremity exoskeletons based on user intention and task-specific movements, the prediction of body joint positions based on EMG features has remained largely unexplored. In this paper, we address this gap by proposing a novel approach that leverages Convolutional Neural Networks and Long-Short-Term Memory (CNN-LSTM) models to generate exoskeleton joint trajectories. Our methodology involves collecting data from three channels of EMG and three degrees-of-freedom (DoF) joint angles and enables us to position control a pneumatic cable-driven upper-limb exoskeleton, thereby assisting users in various tasks. Through extensive experimentation, our intention-based model demonstrates robust performance across different speeds and is capable of detecting variations in payload and electrode placement. The empirical results yielded from our study underscore the efficacy of our approach, particularly in reducing the EMG levels of the user during different tasks by providing exoskeleton assistance as needed.

Keywords: Prosthetics and Exoskeletons, Deep Learning Methods
Abstract: It is essential to accurately identify gait phases when active exoskeleton devices assist with the lower limbs. This work focuses on IMU-based phase detection for stair ambulation. In order to enhance the detection sensitivity of phase transition, this work utilises the LSTM-CRF hybrid model. Four IMU sensors attached to the thighs and shanks on both legs were utilised to collect data during trials on ten healthy subjects for stair ascent and descent. The network’s performance is evaluated by F1-score, recall (true positive rate), and precision, which are 96.3% on average with a standard deviation (std) of 1.9%, 96.6% on average with an std of 1.6%, and 95.9% on average with an std of 2.7%, respectively.

Keywords: Prosthetics and Exoskeletons, Human-Centered Robotics
Abstract: Research in powered prosthesis control has explored the use of impedance-based control algorithms due to their biomimetic capabilities and intuitive structure. Modern impedance controllers feature parameters that smoothly vary over gait phase and task according to a data-driven model. However, these recent efforts only use continuous impedance control during stance and instead utilize discrete transition logic to switch to kinematic control during swing, necessitating two separate models for the different parts of the stride. In contrast, this paper presents a controller that uses smooth impedance parameter trajectories throughout the gait, unifying the stance and swing periods under a single, continuous model. Furthermore, this paper proposes a basis model to represent intertask relationships in the impedance parameters—a strategy that has previously been shown to improve model accuracy over classic linear interpolation methods. In the proposed controller, a weighted sum of Fourier series is used to model the impedance parameters of each joint as continuous functions of gait cycle progression and task. Fourier series coefficients are determined via convex optimization such that the controller best reproduces the joint torques and kinematics in a reference able-bodied dataset. Experiments with a powered knee-ankle prosthesis show that this simpler, unified model produces competitive results when compared to a more complex hybrid impedance-kinematic model over varying walking speeds and inclines.

Keywords: Prosthetics and Exoskeletons, Mechanism Design, Human-Centered Robotics
Abstract: Inspired by the bionic characteristics of ankle and calf skeletal muscles, a novel ankle-foot prosthesis (AFP) with variable stiffness mechanisms (VSMs) is proposed to assist transtibial amputees to restore ankle plantarflexion- dorsiflexion. The prosthesis is designed in the form of a spring-loaded three-loop linkage for function of continuous energy absorption-release in gait stance phase, which can facilitate ankle plantarflexion- dorsiflexion and keep human body move forward steadily. A compliant crank slider mechanism is also developed to power-assist AFP mechanism to improve the adaptive compliant contact between prosthesis and ground. In this paper, mechanics models of the ATP are developed to reveal the variable moment of the ankle joint, which is verified by human-machine simulation. An AFP prototype is built to validate the design experimentally. The results demonstrate that the AFP mechanism has the advantages of low power consumption, human-like joint moment profile. In particular, it is shown that the AFP mechanism with 54W power provided in toe-off phase can reduce the peak power of the motor by 24%.

Keywords: Prosthetics and Exoskeletons, Physical Human-Robot Interaction, Physically Assistive Devices
Abstract: Exoskeleton technologies have numerous potential applications, ranging from improving human motor skills to aiding individuals in their daily activities. While exoskeletons are increasingly viewed, for example, as promising tools in industrial ergonomics, the effect of using them on human motor control, particularly on inter-joint coordination, remains relatively uncharted. This paper investigates the effects of generic low-amplitude force fields applied by an exoskeleton on motor strategies in asymptomatic users. The force fields mimic common perturbations encountered in exoskeletons, such as residual friction, over/under-tuned assistance, or structural elasticity. Fifty-five participants performed reaching tasks while connected to an arm exoskeleton, experiencing one of five tested force fields. Their movements before and after exposure to the exoskeleton force field were compared. The study focuses both on spatial and temporal changes in coordination using specific metrics. The results reveal that even brief exposure to a low-amplitude force field, or to uncompensated residual friction and dynamic forces, applied at the joint level can alter the inter-joint coordination, while task performance remains unaffected. The tested force fields induced varying degrees of changes in joint contributions and synchronization. This study highlights the importance of monitoring coordination changes to fully understand the impact of exoskeletons on human motor control and thus enable safe and widespread adoption of those devices.

Keywords: Prosthetics and Exoskeletons, Physically Assistive Devices, Neurorobotics
Abstract: The vague interpretation of myoelectrical signals on the residual limb end makes restoring dexterous hand function in amputees still impossible. Understanding motor control between human motion intention and synaptic inputs to motor neurons also remains a significant challenge. The neural decoding methods of surface EMG signals remains challenging, which limit the application of robot hand in real life. Herein, we propose and substantiate a human-machine interface for motor control that introduces neural information of motor neurons in conjunction with the combination mechanism of muscle contraction. The interface firstly introduces a new concept of motor unit (MU) spike trains, which combines decoupling of the electrical activations on motor neuron axons with extraction of motion patterns from the discharge timings of the motor neuron pools. We realized a real-time implementation of the EMG decomposition algorithm on our developed prosthesis hand control system. The control scheme provides an accurate classification of intuitive hand motions, enabling the amputee to perform versatile finger movements of the prosthesis hand. The concept of motor neuron discharge timings was evaluated through experiments on one amputee participant and six able-bodied participants. The results show that the neuroprosthesis hand control scheme based on MU spike trains has the capacity of generating accurate and intuitive hand movements for amputees in a physical environment.

Keywords: Prosthetics and Exoskeletons, Product Design, Development and Prototyping, Physical Human-Robot Interaction, Force Control
Abstract: Compared to conventional back-support exoskeletons (BSEs) with two motors, BSEs driven by a single motor have the advantage of light weight. However, current single-motor BSEs have problems with accommodating asynchronous hip movements, achieving precise force control and efficient force transmission, and giving autonomy to users when walking. In this paper, we propose a novel BSE with a differential

series elastic actuator (D-SEA) for lifting assistance. The unique differential working principle can accommodate the angular difference between the hip joints and provide the same assistive torque at both hip joints. The D-SEA achieves precise force control with a custom controller based on accurate spring deflection feedback, and drives the

hip joints via an efficient cable-roller mechanism. Taking advantage of

the active backdrivability of the D-SEA, we proposed an intelligent assistive strategy that automatically provides adequate support for lifting tasks and grants autonomy to users during walking. In the experiments, the BSE reduced the activation level of the back muscles by up to 40% during lifting, without increasing the muscle activation during walking.

Keywords: Human Detection and Tracking, Robot Audition
Abstract: How easy is it to sneak up on a robot? We examine whether we can detect where people are using only the incidental sounds they produce as they move, even when they try to be quiet. We collect a robotic dataset of high-quality 4-channel audio paired with 360 degree RGB data of people moving in different indoor settings. We train models that predict whether there is a moving person nearby and then their location. We implement our method on a robot in real time, demonstrating the ability for robots to navigate populated indoor spaces in a passive manner. For demonstration videos, see our project page: https://sites.google.com/view/unkidnappable-robot

Keywords: Multi-Modal Perception for HRI, Social HRI
Abstract: For a service robot, it is crucial to perceive as early as possible that an approaching person intends to interact: in this case, it can proactively enact friendly behaviors that lead to an improved user experience. We solve this perception task with a sequence-to-sequence classifier of a potential user intention to interact, which can be trained in a self-supervised way. Our main contribution is a study of the benefit of features representing the person’s gaze in this context. Extensive experiments on a novel dataset show that the inclusion of gaze cues significantly improves the classifier performance (AUROC increases from 84.5 % to 91.2 %); the distance at which an accurate classification can be achieved improves from 2.4 m to 3.2 m. We also quantify the system’s ability to adapt to new environments without external supervision. Qualitative experiments show practical applications with a waiter robot.

Keywords: Gesture, Posture and Facial Expressions, Human-Robot Collaboration, Human-Robot Teaming
Abstract: When integrating robots into human daily life, persuasive power can be essential. However, there are often group dynamics which can complicate persuasion. This study focuses on how non-verbal cues, specifically gaze and hand gestures, affect the persuasiveness of a social robot. We have designed a protocol to include non-verbal cues in the social robot Vizzy (head and eye gaze, hand gestures) and test them in a series of experiments using the paradigm of the "Desert Survival Challenge". The goal of the robot is to persuade the participants of the game into changing their answers whilst avoiding negative feelings. It is hypothesized that the non-verbal cues will help avoid psychological reactance without diminishing compliance to the verbal requests issued by the robot. This phenomenon has been verified before for single person persuasion, but it is yet to be tested on groups. Thus, the goal of this project is to verify the effect of non-verbal cues in group persuasion by a robot and comparing it to single person persuasion. The results showed that the robot's gestures increased compliance by the group and the gaze behaviour decreased psychological reactance.

Keywords: Multi-Modal Perception for HRI, Human-Robot Collaboration, Human-Centered Robotics
Abstract: Robot-human object handover has been extensively studied in recent years for a wide range of applications. However, it is still far from being as natural as human-human handovers, largely due to the robots’ limited sensing capabilities. Previous approaches in the literature typically simplify the handover scenarios, including one or more of (a) conducting handovers at fixed locations, (b) not adapting to human preferences, or (c) only focusing on single-arm handover with small objects due to the sensor occlusions caused by large objects. To advance the state of the art toward a human-human level of handover fluency, this paper investigates a bimanual handover scenario in a naturalistic, complex setup. Specifically, we target robot-to-human box transfer while the human partner is on a ladder, and ensure that object is adaptively delivered based on human preferences. To address the occlusion problem that arises in a complex environment, we develop an onboard multi-sensor perception system for the bimanual robot, introduce a measurement confidence estimation technique, and propose an occlusion-resilient multi-sensor fusion technique by positioning visual perception sensors in distinct locations on the robot with different fields of view. Multi-sensor fusion approach achieved a handover success rate above 86.7% for all experiments by successfully combining the strengths of both fields of view of human pose tracking under partial occlusions without sacrificing handover duration.

Keywords: Multi-Modal Perception for HRI, Human-Centered Robotics, Autonomous Agents
Abstract: The socially-aware navigation system has evolved to adeptly avoid various obstacles while performing multiple tasks, such as point-to-point navigation, human-following, and -guiding. However, a prominent gap persists: in Human-Robot Interaction (HRI), the procedure of communicating commands to robots demands intricate mathematical formulations. Furthermore, the transition between tasks does not quite possess the intuitive control and user-centric interactivity that one would desire. In this work, we propose an LLM-driven interactive multimodal multitask robot navigation framework, termed LIM2N, to solve the above new challenge in the navigation field. We achieve this by first introducing a multimodal interaction framework where language and hand-drawn inputs can serve as navigation constraints and control objectives. Next, a reinforcement learning agent is built to handle multiple tasks with the received information. Crucially, LIM2N creates smooth cooperation among the reasoning of multimodal input, multitask planning, and adaptation and processing of the intelligent sensing modules in the complicated system. Extensive experiments are conducted in both simulation and the real world demonstrating that LIM2N has superior user needs understanding, alongside an enhanced interactive experience.

Keywords: Multi-Modal Perception for HRI, Haptics and Haptic Interfaces, Touch in HRI
Abstract: To foster an immersive and natural human-robot interaction (HRI), the implementation of tactile perception and feedback becomes imperative, effectively bridging the conventional sensory gap. In this paper, we propose a dual-modal electronic skin (e-skin) that integrates magnetic tactile sensing and vibration feedback for enhanced HRI. The dual-modal tactile e-skin offers multi-functional tactile sensing and programmable haptic feedback, underpinned by a layered structure comprised of flexible magnetic films, soft silicone elastomer, a Hall sensor and actuator array, and a microcontroller unit. The e-skin captures the magnetic field changes caused by subtle deformations through Hall sensors, employing deep learning for accurate tactile perception. Simultaneously, the actuator array generates mechanical vibrations to facilitate haptic feedback, delivering diverse mechanical stimuli. Notably, the dual-modal e-skin is capable of transmitting tactile information bidirectionally, enabling object recognition and fine-weighing operations. This bidirectional tactile interaction framework will enhance the immersion and efficiency of interactions between humans and robots.

Keywords: Human Detection and Tracking, Sensor-based Control, Rehabilitation Robotics
Abstract: Cane-type robots have been utilized to assist the mobility-impaired population. The essential technique for cane-type robots is to follow the user's ambulation at a close range. This study developed a new cane-type wheeled robot and proposed a novel human-following control frame with multi-camera fusion. This human following control adopts a cascade control strategy consisting of two parts: 1) a human following position controller that locates a user by detecting his/her legs' positions via multi-camera fusion and 2) a cane robot velocity controller to steer the cane robot to the target position. The proposed strategy's effectiveness has been validated in outdoor experiments with six healthy subjects. The experimental scenarios included different terrains (i.e., straight, turning, and inclined paths), road conditions (i.e., flat and rough roads), and walking speeds. The obtained results showed that the average tracking error in the X and Y directions was less than 4.1 cm and 4.4 cm, respectively, and the error in angle was less than 12.9° across all scenarios. Moreover, the cane robot can effectively adapt to a wide range of individual gait patterns and achieve stable human following at daily walking speeds (0.74 m/s - 1.47 m/s).

Keywords: Gesture, Posture and Facial Expressions, Visual Learning, Deep Learning for Visual Perception
Abstract: Interactive hand mesh reconstruction from single-view images poses a significant challenge with the severe occlusion and depth ambiguity inherent in interactive hand gestures. Recent approaches that employ probabilistic models and token-pruned techniques have shown decent results in multi-view human body reconstruction. Nevertheless, these methods have not fully utilized multi-scale semantic information from multi-view images and are not applicable in scenarios involving severe occlusion during dual-hand interactions. Simultaneously, current single-view methods independently reconstruct the left and right hands, which are ineffective in enhancing the interaction between both hands. To address these challenges, we propose CAMInterHand, a cooperative attention-based method for multi-view interactive hand pose and mesh reconstruction. Specifically, CAMInterHand extracts local pyramid features and global vertex features from multi-scale feature maps of multi-view images, enabling the exploration of rich local semantic information and facilitating effective feature alignment. Furthermore, CAMInterHand employs the cooperative attention fusion module to fuse all features from multi-view images, enhancing interactions among vertices of dual hands within global and local contexts. We conduct extensive experiments on the large-scale multi-view dataset InterHand2.6M and CAMInterHand achieves a substantial performance improvement over existing methods for multi-view and single-view interactive hand reconstruction.

Keywords: Multi-Modal Perception for HRI, Gesture, Posture and Facial Expressions, Deep Learning for Visual Perception
Abstract: This paper presents a 2D skeleton-based action segmentation method with applications in fine-grained human activity recognition. In contrast with state-of-the-art methods which directly take sequences of 3D skeleton coordinates as inputs and apply Graph Convolutional Networks (GCNs) for spatiotemporal feature learning, our main idea is to use sequences of 2D skeleton heatmaps as inputs and employ Temporal Convolutional Networks (TCNs) to extract spatiotemporal features. Despite lacking 3D information, our approach yields comparable/superior performances and better robustness against missing keypoints than previous methods on action segmentation datasets. Moreover, we improve the performances further by using both 2D skeleton heatmaps and RGB videos as inputs. To our best knowledge, this is the first work to utilize 2D skeleton heatmap inputs and the first work to explore 2D skeleton+RGB fusion for action segmentation.

Keywords: Force and Tactile Sensing
Abstract: Tactile sensing is significant for robotics since it can obtain physical contact information during manipulation. To capture multimodal contact information within a compact framework, we designed a novel sensor called ViTacTip, which seamlessly integrates both tactile and visual perception capabilities into a single, integrated sensor unit. ViTacTip features a transparent skin to capture fine features of objects during contact, which can be known as the see-through-skin mechanism. In the meantime, the biomimetic tips embedded in ViTacTip can amplify touch motions during tactile perception. For comparative analysis, we also fabricated a ViTac sensor devoid of biomimetic tips, as well as a TacTip sensor with opaque skin. Furthermore, we develop a Generative Adversarial Network (GAN)-based approach for modality switching between different perception modes, effectively alternating the emphasis between vision and tactile perception modes. We conducted a performance evaluation of the proposed sensor across three distinct tasks: i) grating identification, ii) pose regression, iii) contact localization and force estimation. In the grating identification task, ViTacTip demonstrated an accuracy of 99.72%, surpassing TacTip, which achieved 94.60%. It also exhibited superior performance in both pose and force estimation tasks with the minimum error of 0.08mm and 0.03N, respectively, in contrast to ViTac's 0.12mm and 0.15N. Results indicate that ViTacTip outperforms single-modality sensors.

Keywords: Force and Tactile Sensing
Abstract: Tactile sensing plays a critical role in enabling robots to interact safely with target objects in dynamic and unstructured environments. While various tactile sensors based on different sensing principles or different sensitive materials have been proposed, the development of flexible large-area tactile sensors for robots is still challenging. In this paper, a novel highly sensitive piezoresistive sponge based on multi-walled carbon nanotubes (MWCNTs) and polyurethane (PU) sponge is fabricated for pressure sensing. The sensing behavior of the piezoresistive sponge was experimentally evaluated, showing high sensitivity and fast response. Based on the piezoresistive sponge, a flexible large-area tactile sensor is designed for distributed force detection with electrical resistance tomography technology. The sensing performance of the sensor is validated by touch location, sensitivity analysis, real-time touch discrimination, and touch modality recognition. The experimental results indicate that the sensor performs well in detecting the position and force of contact in a large area. The sensor's performance shows promise in embodied tactile sensing and human–robot interaction.

Keywords: Force and Tactile Sensing, Bioinspired Robot Learning, Neurorobotics
Abstract: Tactile exploration of surfaces is a key component of everyday life, allowing us to make complex inferences about our environments even when vision is occluded. The emergence of biomimetic neuromorphic hardware in recent years has furthered our ability to create biologically plausible sensing solutions. While these platforms continue to improve in regards to latency and power consumption, within recent literature on tactile texture classification there is an emphasis on accuracy at the expense of real-time processing. In order for these tactile sensing systems to find use outside of experimental laboratory environments, it is key to design systems capable of capturing and processing data in real-time. Within this paper we present a system for the real-time classification of texture using a neuromorphic tactile sensor, a spiking neural network and a novel decision making algorithm. Our real-time system achieves classification accuracies of 94% on a dataset of 11 natural textile textures. Furthermore our system is capable of identifying textures at human-level performance in as little as 84ms. Additionally, benchmarking our system across CPU, GPU and Loihi2 hardware platforms resulted in a 96% reduction in power consumption on the neuromorphic platform. This system out-performed previous work by the authors and the state of art, both in terms of accuracy and classification speed.

Keywords: Force and Tactile Sensing, Compliant Assembly
Abstract: Tactile sensing has become a popular sensing modality for robot manipulators, due to the promise of providing robots with the ability to measure the rich contact information that gets transmitted through its sense of touch. Among the diverse range of information accessible from tactile sensors, torques transmitted from the grasped object to the fingers through extrinsic environmental contact may be particularly important for tasks such as object insertion. However, tactile torque estimation has received relatively little attention when compared to other sensing modalities, such as force, texture, or slip identification. In this work, we introduce the notion of the Tactile Dipole Moment, which we use to estimate tilt torques from gel-based visuotactile sensors. This method does not rely on deep learning, sensor-specific mechanical, or optical modeling, and instead takes inspiration from electromechanics to analyze the vector field produced from 2D marker displacements. Despite the simplicity of our technique, we demonstrate its ability to provide accurate torque readings over two different tactile sensors and three object geometries, and highlight its practicality for the task of USB stick insertion with a compliant robot arm. These results suggest that simple analytical calculations based on dipole moments can sufficiently extract physical quantities from visuotactile sensors.

Keywords: Force and Tactile Sensing, Dexterous Manipulation, Reinforcement Learning
Abstract: Object pushing presents a key non-prehensile manipulation problem that is illustrative of more complex robotic manipulation tasks. While deep reinforcement learning (RL) methods have demonstrated impressive learning capabilities using visual input, a lack of tactile sensing limits their capability for fine and reliable control during manipulation. Here we propose a deep RL approach to object pushing using tactile sensing without visual input, namely tactile pushing. We present a goal-conditioned formulation that allows both model-free and model-based RL to obtain accurate policies for pushing an object to a goal. To achieve real-world performance, we adopt a sim-to-real approach. Our results demonstrate that it is possible to train on a single object and a limited sample of goals to produce precise and reliable policies that can generalize to a variety of unseen objects and pushing scenarios without domain randomization. We experiment with the trained agents in harsh pushing conditions, and show that with significantly more training samples, a model-free policy can outperform a model-based planner, generating shorter and more reliable pushing trajectories despite large disturbances. The simplicity of our training environment and effective real-world performance highlights the value of rich tactile information for fine manipulation.

Keywords: Force and Tactile Sensing, Haptics and Haptic Interfaces, Machine Learning for Robot Control
Abstract: In this paper, we propose X-lateral teleoperation: a novel hybrid unilateral-bilateral teleoperation framework. Bilateral teleoperation enables kinesthetic coupling between the operator and the remote environment with haptic feedback. However, in free motion, unlike unilateral teleoperators, bilateral teleoperators reflect undesirable operational forces to the operator. The proposed X-lateral teleoperation framework benefits from a learning-based contact detection algorithm which triggers switching from unilateral teleoperation in free motion to bilateral teleoperation in contact. We also present a neural network based two-class classification technique to detect contacts even with environments not seen in training. In experiments with linear motors and Phantom Omni devices, using sensorless force estimation, we show that the proposed method can decrease operational forces significantly over transparency-optimized bilateral architectures.

Keywords: Force and Tactile Sensing, Mechanism Design
Abstract: GelSight sensors that estimate contact geometry and force by reconstructing the deformation of their soft elastomer from images would yield poor force measurements when the elastomer deforms uniformly or reaches deformation saturation. Here we present L3 F-TOUCH sensor that considerably enhances the three-axis force sensing capability of typical GelSight sensors. Specifically, L3 F-TOUCH sensor comprises: (i) an elastomer structure resembling the classic GelSight sensor design for fine-grained contact geometry sensing; and (ii) a mechanically simple suspension structure to enable three-dimensional elastic displacement of the elastomer structure upon contact. Such displacement is tracked by detecting the displacement of an ARTag and is transformed to three-axis contact force via calibration. We further revamp the sensor’s optical system by fixing the ARTag on the base and reflecting it to the same camera viewing the elastomer through a mirror. As a result, the tactile and force sensing modes can operate independently, but the entire L3 F-TOUCH remains Light-weight and Low-cost while facilitating a wireLess deployment. Evaluations and experiment results demonstrate that the proposed L3 F-TOUCH sensor compromises GelSight’s limitation in force sensing and is more practical compared with equipping commercial three-axis force sensors. Thus, the L3 F-TOUCH could further empower existing Vision-based Tactile Sensors (VBTSs) in replication and deployment.

Keywords: Force and Tactile Sensing, Mechanism Design, Grippers and Other End-Effectors
Abstract: Compared to fully-actuated robotic end-effectors, underactuated ones are generally more adaptive, robust, and cost-effective. However, state estimation for underactuated hands is usually more challenging. Vision-based tactile sensors, like Gelsight, can mitigate this issue by providing high-resolution tactile sensing and accurate proprioceptive sensing. As such, we present GelLink, a compact, underactuated, linkage-driven robotic finger with low-cost, high-resolution vision-based tactile sensing and proprioceptive sensing capabilities. In order to reduce the amount of embedded hardware, i.e. the cameras and motors, we optimize the linkage transmission with a planar linkage mechanism simulator and develop a planar reflection simulator to simplify the tactile sensing hardware. As a result, GelLink only requires one motor to actuate the three phalanges, and one camera to capture tactile signals along the entire finger. Overall, GelLink is a compact robotic finger that shows adaptability and robustness when performing grasping tasks. The integration of vision-based tactile sensors can significantly enhance the capabilities of underactuated fingers and potentially broaden their future usage.

Keywords: Force and Tactile Sensing, Mechanism Design, Soft Sensors and Actuators
Abstract: Camera-based tactile sensors can provide high resolution positional and local geometry information for robotic manipulation. Curved and rounded fingers are often advantageous, but it can be difficult to derive illumination systems that work well within curved geometries. To address this issue, we introduce RainbowSight, a family of curved, compact, camera-based tactile sensors which use addressable RGB LEDs illuminated in a novel rainbow spectrum pattern. In addition to being able to scale the illumination scheme to different sensor sizes and shapes to fit on a variety of end effector configurations, the sensors can be easily manufactured and require minimal optical tuning to obtain high resolution depth reconstructions of an object deforming the sensor’s soft elastomer surface. Additionally, we show the advantages of our new hardware design and improvements in calibration methods for accurate depth map generation when compared to alternative lighting methods commonly implemented in previous camera-based tactile sensors. With these advancements, we make the integration of tactile sensors more accessible to roboticists by allowing them the flexibility to easily customize, fabricate, and calibrate camera-based tactile sensors to best fit the needs of their robotic systems.

Keywords: Humanoid and Bipedal Locomotion, Formal Methods in Robotics and Automation, Collision Avoidance
Abstract: This study proposes a novel planning framework based on a model predictive control formulation that incorporates signal temporal logic (STL) specifications for task completion guarantee and robustness quantification. This marks the first-ever study to apply STL-guided trajectory optimization for bipedal locomotion push recovery, where the robot experiences unexpected disturbances. Existing recovery strategies often struggle with complex task logic reasoning and locomotion robustness evaluation, making them susceptible to failures due to inappropriate recovery strategies or insufficient robustness. To address this issue, the STL-guided framework generates optimal and safe recovery trajectories that simultaneously satisfy the task specification and maximize the locomotion robustness. Our framework outperforms a state-of-the-art locomotion controller in a high-fidelity dynamic simulation, especially in scenarios involving crossed-leg maneuvers. Furthermore, it demonstrates versatility in tasks such as locomotion on stepping stones, where the robot must select from a set of disjointed footholds to maneuver successfully.

Keywords: Humanoid and Bipedal Locomotion, Legged Robots, Humanoid Robot Systems
Abstract: This paper presents a reactive locomotion method for bipedal robots enhancing robustness and external disturbance rejection performance by seamlessly rendering several walking strategies of the ankle, hip, and footstep adjustment. The Nonlinear Model Predictive Control (NMPC) is formulated to take into account nonlinear Divergent Component of Motion (DCM) error dynamics that predicts the future states of the robot in response to the walking strategies. This formulated NMPC enables the seamless application of these strategies improving push disturbance rejection performance. The proposed controller is validated in simulation and through an experiment on a bipedal robot platform, Gazelle, which confirms its effectiveness in real-time.

Keywords: Humanoid and Bipedal Locomotion, Legged Robots, Hybrid Logical/Dynamical Planning and Verification
Abstract: Successfully achieving bipedal locomotion remains challenging due to real-world factors such as model uncertainty, random disturbances, and imperfect state estimation. In this work, we propose a novel metric for locomotive robustness -- the estimated size of the hybrid forward invariant set associated with the step-to-step dynamics. Here, the forward invariant set can be loosely interpreted as the region of attraction for the discrete-time dynamics. We illustrate the use of this metric towards synthesizing nominal walking gaits using a simulation-in-the-loop learning approach. Further, we leverage discrete-time barrier functions and a sampling-based approach to approximate sets that are maximally forward invariant. Lastly, we experimentally demonstrate that this approach results in successful locomotion for both flat-foot walking and multi-contact walking on the Atalante lower-body exoskeleton.

Keywords: Humanoid and Bipedal Locomotion, Legged Robots, Whole-Body Motion Planning and Control
Abstract: For humans, fast, efficient walking over flat ground represents the vast majority of locomotion that an individual experiences on a daily basis, and for an effective, real-world humanoid robot the same will likely be the case. In this work, we propose a locomotion controller for efficient walking over near-flat ground using a relatively simple, model-based controller that utilizes a novel combination of several interesting design features including an ALIP-based step adjustment strategy, stance leg length control as an alternative to center of mass height control, and rolling contact for heel-to-toe motion of the stance foot. We then present the results of this controller on our robot Nadia, both in simulation and on hardware. These results include validation of this controller’s ability to perform fast, reliable forward walking at 0.75 m/s along with backwards walking, side-stepping, turning in place, and push recovery. We also present an efficiency comparison between the proposed control strategy and our baseline walking controller over three steady-state walking speeds. Lastly, we demonstrate some of the benefits of utilizing rolling contact in the stance foot, specifically the reduction of necessary positive and negative work throughout the stride.

Keywords: Humanoid and Bipedal Locomotion, Motion and Path Planning, Humanoid Robots, Legged Robots
Abstract: Running and jumping are locomotion modes that allow legged robots to rapidly traverse great distances and overcome difficult terrain. In this article, we show that the 3-D Divergent Component of Motion (3D-DCM) framework, which was successfully used for generating walking trajectories in previous works, retains its validity and coherence during flight phases, and, therefore, can be used for planning running and jumping motions. We propose a highly efficient motion planner that generates stable center-of-mass (CoM) trajectories for running and jumping with arbitrary contact sequences and time parametrizations. The proposed planner constructs the complete motion plan as a sequence of motion phases that can be of different types: stance, flight, transition phases, etc. We introduce a unified formulation of the CoM and DCM waypoints at the start and end of each motion phase, which makes the framework extensible and enables the efficient waypoint computation in matrix and algorithmic form. The feasibility of the generated reference trajectories is demonstrated by extensive whole-body simulations with the humanoid robot TORO.

Keywords: Humanoid and Bipedal Locomotion, Representation Learning, Legged Robots
Abstract: This paper presents a novel framework for learning robust bipedal walking by combining a data-driven state representation with a Reinforcement Learning (RL) based locomotion policy. The framework utilizes an autoencoder to learn a low-dimensional latent space that captures the complex dynamics of bipedal locomotion from existing locomotion data. This reduced dimensional state representation is then used as states for training a robust RL-based gait policy, eliminating the need for heuristic state selections or the use of template models for gait planning. The results demonstrate that the learned latent variables are disentangled and directly correspond to different gaits or speeds, such as moving forward, backward, or walking in place. Compared to traditional template model-based approaches, our framework exhibits superior performance and robustness in simulation. The trained policy effectively tracks a wide range of walking speeds and demonstrates good generalization capabilities to unseen scenarios.

Keywords: Humanoid Robot Systems, Legged Robots, Sensor Fusion
Abstract: High-frequency and accurate state estimation is crucial for biped robots. This paper presents a tightly-coupled LiDAR-Inertial-Kinematic Odometry (LIKO) for biped robot state estimation based on an iterated extended Kalman filter. Beyond state estimation, the foot contact position is also modeled and estimated. This allows for both position and velocity updates from kinematic measurement. Additionally, the use of kinematic measurement results in an increased output state frequency of about 1kHz. This ensures temporal continuity of the estimated state and makes it practical for control purposes of biped robots. We also announce a biped robot dataset consisting of LiDAR, inertial measurement unit (IMU), joint encoders, force/torque (F/T) sensors, and motion capture ground truth to evaluate the proposed method. The dataset is collected during robot locomotion, and our approach reached the best quantitative result among other LIO-based methods and biped robot state estimation algorithms. The dataset and source code will be available at https://github.com/Mr-Zqr/LIKO.

Keywords: Legged Robots, Actuation and Joint Mechanisms, Mechanism Design
Abstract: This paper introduces Barry, a dynamically balancing quadruped robot optimized for high payload capabilities and efficiency. It presents a new high-torque and low-inertia leg design, which includes custom-built

high-efficiency actuators and transparent, sensorless transmissions. The robot’s reinforcement learning-based controller is trained to fully

leverage the new hardware capabilities to balance and steer the robot. The newly developed controller can manage the non-linearities introduced by the new leg design and handle unmodeled payloads up to 90kg while operating at high efficiency. The approach’s efficacy is demonstrated by a high payload-to-weight ratio verified with multiple tests, with a maximum ratio of 2 on flat terrain. Experiments also demonstrate Barry’s power consumption and cost of transport, which converge to a value of 0.7 at 1.4m/s, regardless of the payload mass.

Keywords: Motion Control, Actuation and Joint Mechanisms, Redundant Robots
Abstract: This letter focuses on the design and control of a novel seven-degree-of-freedom (7-DOF) robotic manipulator (D-Arm) to address the issue of backlash nonlinearity coupling unknown disturbance through the dual-motor anti-backlash control technology. Specifically, the first three axes of the D-Arm near the base are implemented as dual-motor joints (DMJs), while the remaining four axes are single-motor joints (SMJs), which achieve a more comprehensive performance. For DMJs, which are over-actuated systems, we first discover an internal disturbance phenomenon named servo-conflict and consider it in the controller design. To mitigate the adverse effects of backlash coupling disturbance, an admittance control-based position compensator is proposed. Then, after the backlash elimination, a dual-motor linear active disturbance rejection controller is effectively developed for load tracking task of the DMJ. In the presence of unknown backlash and disturbance, the proposed strategy can improve both transient and steady-state position tracking response, reduce energy consumption without requiring any backlash model information. The effectiveness and simplicity of the developed control strategy are verified through comparative experiments on the D-Arm.

Keywords: Motion Control, Aerial Systems: Applications
Abstract: This paper presents a novel solution for UAV control in cooperative multi-robot systems, which can be used in various scenarios such as leader-following, landing on a moving base, or specific relative motion with a target. Unlike classical methods that tackle UAV control in the world frame, we directly control the UAV in the target coordinate frame, without making motion assumptions about the target. In detail, we formulate a non- linear model predictive controller of a UAV, referred to as the agent, within a non-inertial frame (i.e., the target frame). The system requires the relative states (pose and velocity), the angular velocity and the accelerations of the target, which can be obtained by relative localization methods and ubiquitous MEMS IMU sensors, respectively. This framework eliminates dependencies that are vital in classical solutions, such as accurate state estimation for both the agent and target, prior knowledge of the target motion model, and continuous trajectory re-planning for some complex tasks. We have performed extensive simulations to investigate the control performance with varying motion characteristics of the target. Furthermore, we conducted real robot experiments, employing either simulated relative pose estimation from motion capture systems indoors or directly from our previous relative pose estimation devices outdoors, to validate the applicability and feasibility of the proposed approach.

Keywords: Motion Control, Aerial Systems: Mechanics and Control, Failure Detection and Recovery, Passive Fault-Tolerant Control
Abstract: This study proposes a uniform passive fault-tolerant control (FTC) method for a quadcopter that does not rely on fault information subject to one, two adjacent, two opposite, or three rotor failure. The uniform control implies that the passive FTC is able to cover the condition from quadcopter fault-free to rotor failure without controller switching. To achieve the purpose of the passive FTC, the rotors’ fault is modeled as a lumped disturbance acting on the virtual control of the quadcopter system. The disturbance estimate is used directly for the passive FTC with rotor failure. At the same time, a modified controller structure is designed to achieve the passive FTC ability for two and three rotor failure. To avoid the control allocation switch from the fault-free control to the FTC, a dynamic control allocation is used. In addition, the closed-loop stability is analyzed under up to three rotor failure. To validate the proposed uniform passive FTC method, outdoor experiments are performed for the first time, which have demonstrated that the hovering quadcopter is able to recover from one rotor failure by the proposed controller and continue to fly even if two adj

Keywords: Motion Control, Aerial Systems: Mechanics and Control, Industrial Robots
Abstract: This work focuses on the agile transportation of liquids with robotic manipulators. In contrast to existing methods that are either computationally heavy, system/container specific or dependant on a singularity-prone pendulum model, we present a real-time slosh-free tracking technique. This method solely requires the reference trajectory and the robot's kinematic constraints to output kinematically feasible joint space commands. The crucial element underlying this approach consists on mimicking the end-effector's motion through a virtual quadrotor, which is inherently slosh-free and differentially flat, thereby allowing us to calculate a slosh-free reference orientation. Through the utilization of a cascaded proportional-derivative (PD) controller, this slosh-free reference is transformed into task space acceleration commands, which, following the resolution of a Quadratic Program (QP) based on Resolved Acceleration Control (RAC), are translated into a feasible joint configuration. The validity of the proposed approach is demonstrated by simulated and real-world experiments on a 7 DoF Franka Emika Panda robot.

Keywords: Motion Control, Autonomous Agents, Optimization and Optimal Control
Abstract: Autonomous vehicle motion control development requires testing and evaluation at all stages of the process. The development phase involving the instrumentation and operation of a full-size vehicle can be especially costly. Scale vehicles have been developed in the literature to serve as a cost-effective transition from testing in a simulation environment to a physical system. However, the existing scale vehicle platforms do not support four-wheel steering and cannot isolate the performance of motion control algorithms from other modules such as perception and path planning. This paper closes this gap by proposing a new scale vehicle platform, called JetRacer-4WS, based on the open-source JetRacer autonomous vehicle with additional modifications to support four-wheel steering, model predictive control-based path following, and high-precision ultrasonic-based real-time positioning. The proposed JetRacer-4WS can be used as a low-cost platform for both research and education on motion control, path following, and vehicle dynamics. We describe the design of JetRacer-4WS, and experimentally demonstrate JetRacer-4WS’ ability to perform controller auto-tuning and illustrate the advantage of four-wheel steering. We also show that JetRacer-4WS can be used as a validation platform for testing advanced control algorithms such as event-triggered model predictive control. The associated code is open-sourced and available at: https://github.com/jchenee2015/jetracer-4WS.

Keywords: Motion Control, Collision Avoidance, Planning under Uncertainty
Abstract: Collision-free navigation is a critical issue in robotic systems as the environment is often dynamic and uncertain. This paper investigates a data-stream-driven motion control problem for mobile robots to avoid randomly moving obstacles when the probability distribution of the obstacle’s movement is partially observable through data and can be even time varying. A data-stream-driven ambiguity set is firstly constructed from movement data by leveraging a Dirichlet process mixture model and is updated online using real-time data. Then we propose an Online-Learning-based Distributionally Robust Nonlinear Model Predictive Control (OL-DR-NMPC) approach for limiting the risk of collision through considering the worst-case distribution within the ambiguity set. To facilitate solving the OL-DR-NMPC problem, we reformulate it as a finite-dimensional nonlinear optimization problem. To cope with the bilinear matrix inequality constraints in the nonlinear problem, we develop a parabolic relaxation and a sequential algorithm, by which the problem is further transformed into polynomial-time solvable surrogates. The simulations using a quadrotor model are employed to demonstrate the effectiveness and advantages of the proposed method.

Keywords: Motion Control, Compliance and Impedance Control, Physical Human-Robot Interaction
Abstract: The presented work tackles the question of quantifying the pose deviations of robots subject to external disturbance forces. While this question may not be central for large robots perfectly rejecting disturbances through high controller gains, it is an important factor when considering collaborative settings where smaller robots may be deviated from their task because of unmodeled physical interactions. This is all the more true with human-robot collaboration where human capacities may fluctuate over time and have to be compensated by a proper adaptation of the robot control. To move forward in this direction, this work first derives a deviation prediction methodology and exemplifies it using three largely employed control approaches. The proposed prediction method is then validated using simulated and real robot experiments both in single and multiple robots cases. The obtained results constitute a stepping stone towards a quantitative metric for robots adapting their behaviour to human motor fluctuation.

Keywords: Motion Control, Flexible Robotics, Parallel Robots
Abstract: This paper proposed an iterative learning control (ILC) scheme for deformable open-frame cable-driven parallel robots (D-CDPRs). In contrast to the straightforward inverse kinematics of the rigid frame cable-driven parallel robots (CDPRs), accurate modeling of the deformable frame poses challenges due to errors and uncertainties. To address these issues, the authors propose the use of ILC, a control strategy that modifies the control input over iterations based on previous results. ILC has been successfully applied to traditional cable robots, particularly in handling model uncertainty. The paper presents a novel ILC control scheme specifically designed for D-CDPRs, with a focus on reducing tracking errors over repetitive operations. Additionally, hardware experiments are conducted to validate the effectiveness and reliability of the proposed ILC approach. The results demonstrate the efficacy of ILC in mitigating tracking errors, even in scenarios where the dynamic model of the D-CDPRs is unknown.

Keywords: Motion Control, Industrial Robots, Human-Robot Collaboration
Abstract: Force-free control (FFC) allows for flexible manipulator motion in response to external forces, making it a vital component of human-robot interaction (HRI). Manual intervention may cause uneven forces on the manipulator or frequencies close to the natural frequency, and mechanical resonance can occur due to the inertia of the manipulator and adjustable equivalent stiffness of the controller. This paper proposes an FFC approach for industrial manipulators using a six-axis force/torque sensor (F/T sensor), implemented through a three-layer control architecture, consisting of motion control layer, Admittance Control layer and force decoupling layer. To mitigate the effects of mechanical resonance, a high-pass filter (HPF) is integrated with the F/T sensor and its impact is investigated. Experimental validation is conducted using both a simulation model and an industrial manipulator. Test results indicate that the proposed FFC architecture enables the manipulator not only to interact smoothly with external forces, but also to distinguish load forces at different frequencies and potentially address the issue of mechanical resonance between the manipulator and the applied load forces.

Keywords: Medical Robots and Systems, Acceptability and Trust, Safety in HRI
Abstract: The injection of therapeutic agents into the subretinal space might allow improved treatment of age-related macular degeneration. Various robotic systems have been developed to achieve the required precision, and in combination with intraoperative Optical Coherence Tomography (iOCT) imaging, methods for autonomous robotic guidance have been proposed. In such systems, the robot’s cognition is often governed by machine learning algorithms, such as convolutional neural networks (CNNs), which provide semantic scene information from iOCT images. Although the robot performs a surgical task autonomously, human supervision is critical to monitor the robot’s execution and, if necessary, stop the robot or take control to avoid trauma to the patient. In this paper, we propose a novel visualization concept for improved human supervision of autonomous robotic subretinal injection that integrates uncertainty information of the data provided to the robot. We design a focus and context visualization that renders an automatically identified instrument-aligned B-scan in the context of the 3D OCT volume. Our visualization is enriched by augmenting the uncertainty information on the instrument-aligned B-scan. To dynamically model task-specific uncertainty, we introduce a weighting scheme to assign an importance factor to each pair of classes, controlling the impact of their confusion on the overall uncertainty. We demonstrate our visualization concept on iOCT volumes acquired at different stages during subretinal injection on ex-vivo porcine eyes. We show that our processing pipeline achieves sufficient update rates for surgical display and discuss the impact of our visualization concept on the acceptance of robotic task autonomy for subretinal injection procedures.

Keywords: Medical Robots and Systems, Automation at Micro-Nano Scales
Abstract: Colon endoscopic robots represent a promising screening modality for the visualization of colon cancers with high sensitivity. However, current colonoscopy robots are often characterized by intricate and bulky mechanical structures, which pose practical challenges when moving through the complex and narrow environment of the colon. Moreover, these robots are typically equipped with a single camera, limiting their ability to accurately estimate the depth of haustral folds in colon, which is of great importance for the active colonic navigation of the robots. To address these challenges, we develop a track-based stereoscopic endoscopic robot (TSER) which is equipped with four tracks positioned at the corners of its body. This innovative design maximizes the contact between the tracks and the colon wall, enhancing maneuverability. The tracks are constructed from de-molded polydimethylsiloxane (PDMS) and incorporate micro-patterns on their outer surfaces. We have proposed a straightforward strategy for detecting haustral folds using TSER's stereo camera, which allows for precise identification of their position and depth. The TSER achieves an average motion speed of 9.8 mm/s in a bellows tube that contains silicone oil and a speed of 5.2 mm/s in an ex-vivo porcine intestinal segment. Impressively, the TSER boasts an 88.11% accuracy rate in haustral fold depth estimation, surpassing the performance of existing geometric shape fitting methods. These results demonstrate that the TSER holds great potential for effective and efficient movement and inspection within the colon, offering a promising solution for improved colon cancer screening.

Keywords: Medical Robots and Systems, Calibration and Identification, Kinematics
Abstract: Telesurgical robotic systems provide a well established form of assistance in the operating theater, with evidence of growing uptake in recent years. Until now, the da Vinci surgical system has been the most widely adopted robot of this kind. To accelerate research on robotic-assisted surgery, the retired first-generation da Vinci robots have been redeployed for research use as "da Vinci Research Kits" (dVRKs), which have been distributed to research institutions around the world. In the past ten years, a great amount of research on the dVRK has been carried out across a vast range of research topics. During this extensive and distributed process, common technical issues have been identified that are buried deep within the dVRK research and development architecture. This paper gathers and analyzes the most significant of these, with a focus on the technical constraints of the first-generation dVRK, which both existing and prospective users should be aware of before embarking onto dVRK-related research. The hope is that this review will aid users in identifying and addressing common limitations of the systems promptly, thus helping to accelerate progress in the field.

Keywords: Medical Robots and Systems, Compliance and Impedance Control, Physical Human-Robot Interaction
Abstract: In the context of telehealth, robotic approaches have proven a valuable solution to in-person visits in remote areas, with decreased costs for patients and infection risks. In particular, in ultrasonography, robots have the potential to reproduce the skills required to acquire high-quality images while reducing the sonographer's physical efforts. In this paper, we address the control of the interaction of the probe with the patient's body, a critical aspect of ensuring safe and effective ultrasonography. We introduce a novel approach based on variable impedance control, allowing the real-time optimisation of compliant controller parameters during ultrasound procedures. This optimisation is formulated as a quadratic programming problem and incorporates physical constraints derived from viscoelastic parameter estimations. Safety and passivity constraints, including an energy tank, are also integrated to minimise potential risks during human-robot interaction. The proposed method's efficacy is demonstrated through experiments on a patient's dummy torso, highlighting its potential for achieving safe behaviour and accurate force control during ultrasound procedures, even in cases of contact loss.

Keywords: Medical Robots and Systems, Compliant Joints and Mechanisms, Surgical Robotics: Steerable Catheters/Needles
Abstract: The paper introduces a novel robotic system for transanal endoscopic microsurgery (TEM) with a master-slave operated configuration. This slave manipulator features a modular distal continuum section, comprising two 7-DoF surgical instruments and a 5-DoF endoscopic arm designed to enhance hand-eye coordination and instrument triangulation in narrow and shallow rectal spaces. Key innovations include the hybrid coaxial continuum unit (HCCU) for improved bending characteristics and structural stiffness, and a design optimization for dexterity under anatomical constraints. Experimental validations demonstrate the system's precision and capability in simulated surgical tasks, highlighting its potential for advanced TEM applications with improved operational dexterity and reduced view obstruction.

Keywords: Medical Robots and Systems, Computer Vision for Medical Robotics, Deep Learning Methods
Abstract: Ultrasound (US) imaging is widely used in diagnosing and staging abdominal diseases due to its lack of non-ionizing radiation and prevalent availability. However, significant inter-operator variability and inconsistent image acquisition hinder the widespread adoption of broader screening programs. Robotic ultrasound systems have emerged as a promising solution, offering standardized acquisition protocols and the possibility of automated acquisition. Additionally, robotic ultrasound systems enable access to 3D data via robotic tracking and incoherent compounding of ultrasound frames, which results in improved interpretation and disease diagnosis. However, the interpretability of 3D ultrasound reconstruction of abdominal images can be affected by the patient's breathing motion. This study introduces a method to compensate for breathing motion in 3D ultrasound compounding by leveraging implicit neural representations. Our approach employs a robotic ultrasound system for automated screenings. To demonstrate the method's effectiveness, we evaluate our proposed method for the diagnosis and monitoring of abdominal aorta aneurysms as a representative use case. Our experiments demonstrate that our proposed pipeline facilitates robust automated robotic acquisition, mitigating artifacts from breathing motion, and yields smoother 3D reconstructions for enhanced screening and medical diagnosis.

Keywords: Medical Robots and Systems, Computer Vision for Medical Robotics, Localization
Abstract: In ophthalmic surgeries, such as vitreoretinal operations, surgeons rely on imaging systems, primarily microscopes, for real-time instrument monitoring and motion planning. However, novice surgeons struggle to estimate instrument positions from 2D microscope frames, necessitating extensive trial-and-error experience with the background that additional imaging modalities such as iOCT remain inaccessible in most operating rooms. Targeting intraocular assessment within the current surgical setup, this paper presents an image-based pose estimation method to obtain real-time instrument positions in a standard 12mm-radius spherical eyeball model, achieved by linking floating instruments with on-the-retinal objects based on the intraocular shadowing principle. We validate this pose-estimation method in a Unity simulator and verify its depth estimation capability using a specially designed eyeball phantom. Both simulator and phantom experiments demonstrate an average needle-tip estimation error within [1.0, 2.0] mm using only 2D microscope frames.

Keywords: Medical Robots and Systems, Computer Vision for Medical Robotics, Robotics and Automation in Life Sciences
Abstract: Autonomous ultrasound (US) scanning has attracted increased attention, and it has been seen as a potential solution to overcome the limitations of conventional US examinations, such as inter-operator variations. However, it is still challenging to autonomously and accurately transfer a planned scan trajectory on a generic atlas to the

current setup for different patients, particularly for thorax applications with limited acoustic windows. To address this challenge, we proposed a skeleton graph-based non-rigid registration to adapt patient-specific properties using subcutaneous bone surface features rather than the skin surface. To this end, the self-organization mapping is successively used twice to unify the input point cloud and extract the key points, respectively. Afterward, the minimal spanning tree is employed to generate a tree graph to connect all extracted key points. To appropriately characterize the rib cartilage outline to match the source and target point cloud, the path extracted from the tree graph is optimized by maximally maintaining continuity throughout each rib. To validate the proposed approach, we manually extract the US

cartilage point cloud from one volunteer and seven CT cartilage point clouds from different patients. The results demonstrate that the proposed graph-based registration is more effective and robust in adapting to the inter-patient variations than the ICP (distance error mean±SD: 5.0±1.9mm vs 8.6±6.7mm on seven CTs).

Keywords: Medical Robots and Systems, Computer Vision for Medical Robotics
Abstract: This study proposes a method for finding a parasternal long-axis view in echocardiography autonomously with a robotic ultrasound (US) system. In obtaining this view, it is necessary to avoid the ribs and lungs because they reduce the clarity of US image. Meanwhile, the anatomical position and size of the heart, lungs, and ribs differ between individuals, which makes it difficult to find the optimal position of the US probe. Our proposed system is comprised of the following three processes. First, an exhaustive scan of the chest wall region is performed. The position of the probe that allows the mitral valve to be centrally positioned is estimated based on this scan. Second, the probe is rotated once in the yaw direction while being fixed in that position. The yaw angle is estimated at a point parallel to the left ventricular longitudinal axis in the acquired images. Finally, the pitch angle of the probe is estimated so that the probe avoids the connection between the mitral valve and the papillary muscle and chordae. To validate the proposed method, we performed human trials with five healthy subjects and measured the detection rate of observation points used to evaluate the image quality of parasternal long-axis view. The result showed that the median detection rate of the observation points was 63.3 ± 5.3%, which implies that the proposed method is valid.