Keywords: AI-Based Methods, Service Robotics, Semantic Scene Understanding
Abstract: The state recognition of the environment and objects by robots is generally based on the judgement of the current state as a classification problem. On the other hand, state changes of food in cooking happen continuously and need to be captured not only at a certain time point but also continuously over time. In addition, the state changes of food are complex and cannot be easily described by manual programming. Therefore, we propose a method to recognize the continuous state changes of food for cooking robots through the spoken language using pre-trained large-scale vision-language models. By using models that can compute the similarity between images and texts continuously over time, we can capture the state changes of food while cooking. We also show that by adjusting the weighting of each text prompt based on fitting the similarity changes to a sigmoid function and then performing black-box optimization, more accurate and robust continuous state recognition can be achieved. We demonstrate the effectiveness and limitations of this method by performing the recognition of water boiling, butter melting, egg cooking, and onion stir-frying.

Keywords: AI-Enabled Robotics, Natural Dialog for HRI, Multi-Contact Whole-Body Motion Planning and Control
Abstract: This paper presents Words2Contact, a language-guided multi-contact placement pipeline leveraging large language models and vision language models. Our method is a key component for language-assisted teleoperation and human-robot cooperation, where human operators can instruct the robots where to place their support contacts before whole-body reaching or manipulation using natural language. Words2Contact transforms the verbal instructions of a human operator into contact placement predictions; it also deals with iterative corrections, until the human is satisfied with the contact location identified in the robot's field of view. We benchmark state-of-the-art LLMs and VLMs for size and performance in contact prediction. We demonstrate the effectiveness of the iterative correction process, showing that users, even naive, quickly learn how to instruct the system to obtain accurate locations. Finally, we validate Words2Contact in real-world experiments with the Talos humanoid robot, instructed by human operators to place support contacts on different locations and surfaces to avoid falling when reaching for distant objects.

Keywords: Gesture, Posture and Facial Expressions, Telerobotics and Teleoperation, Haptics and Haptic Interfaces
Abstract: Accurate sensing equipment for capturing human hand data is crucial for robot hand teleoperation. Individual calibration processes to reflect individual anatomical differences in human hands are complex and time-consuming. Additionally, handling noise caused by slipping or impact while wearing sensing equipment is challenging. This paper proposes an efficient calibration system integrating Kernel Principal Component Analysis (Kernel PCA) with hand synergy to overcome these limitations. By utilizing Kernel PCA, the proposed approach enables the reconstruction of both object-grasping and non-grasping hand postures based on human hand synergy. The accuracy of the reconstructed hand postures is evaluated using four principal components (PCs) and comparing data from five different users. The proposed method improves accuracy by 5~degrees and reduces the standard deviation by 3~degrees compared to traditional hand synergy using Principal Component Analysis (PCA). Kernel PCA demonstrates high robustness to noise from sensing equipment, ensuring reliable hand posture reproduction under various conditions. The application of this system to an actual robot hand verifies its practical utility, providing reliable control across various users and scenarios.

Keywords: Humanoid Robot Systems, Compliance and Impedance Control, Compliant Joints and Mechanisms
Abstract: In this work, the novel, open-source humanoid robot, PANDORA, is presented where a majority of the structural elements are manufactured using 3D-printed compliant materials. As opposed to contemporary approaches that incorporate the elastic element into the actuator mechanisms, PANDORA is designed to be compliant under load, or in other words, structurally elastic. This design approach lowers manufacturing cost and time, design complexity, and assembly time while introducing controls challenges in state estimation, joint and whole-body control. This work features an in-depth description on the mechanical and electrical subsystems including details regarding additive manufacturing benefits and drawbacks, usage and placement of sensors, and networking between devices. In addition, the design of structural elastic components and their effects on overall performance from an estimation and control perspective are discussed. Finally, results are presented which demonstrate the robot completing a robust balancing objective in the presence of disturbances and stepping behaviors.

Keywords: Actuation and Joint Mechanisms, Compliant Joints and Mechanisms, Legged Robots
Abstract: Robots avoid collisions as these can lead to hardware damage. To deploy robots in less structured environments, robust hardware is required. Traditional approaches that improve impact-resilience result in bulky actuators, increased backlash or decreased torque density. A lightweight and compact clutch principle based on wedged rollers is presented as well as an experimental proof-of-concept. A use case involving the design of clutched planetary gearbox actuators tailored to humanoid robot legs shows a reduction in axial length of 25% up to 40% depending on the maximum expected overload.

Keywords: Human-Aware Motion Planning, Human-Centered Automation, Collision Avoidance
Abstract: Social navigation in robotics primarily involves guiding mobile robots through human-populated areas, with pedestrian comfort balanced with efficient path-finding. Although progress has been seen in this field, a solution for the seamless integration of robots into pedestrian settings remains elusive. In this paper, a social force model for legged robots is developed, utilizing visual perception for human localization. In particular, an augmented social force model is introduced, incorporating refined interpretations of repulsive forces and avoidance behaviors based on pedestrian actions, alongside a target following mechanism. Experimental evaluation on a quadruped robot, through various scenarios, including interactions with oncoming pedestrians, crowds, and obstructed paths, demonstrates that the proposed augmented model significantly improves upon previous baseline methods in terms of chosen path length, average velocity, and time-to-goal for effective and efficient social navigation. The code is open-source, while video demonstrations can be found on the project's webpage: https://rpl-cs-ucl.github.io/ASFM

Keywords: Gesture, Posture and Facial Expressions, Human-Centered Robotics, Social HRI
Abstract: In human-human interaction, posture serves as a critical non-verbal cue that subconsciously shapes first impressions and perceptions. Given the preference for anthropomorphism of robots acting in socially intensive situations such as caregiving, the influence of posture is expected to increase in interactions with humanoid robots. This study aims to identify the most preferred default position for the assistive humanoid robot GARMI, ensuring users’ perceived safety in human-robot interaction (HRI). In a preliminary study, 30 participants evaluated ten different arm postures of the robot GARMI regarding their perceived discomfort. From these evaluations and direct rankings, three arm positions were selected for further analysis alongside the current default position in the virtual reality (VR) study. In this subsequent study, 50 participants assessed their perception of safety using both objective measures of comfort distance, grounded in proxemic theory, and two subjective measures, i.e., Godspeed questionnaire and Robotic Social Attribute Scale (RoSaS). The results indicate a significant impact of the robot’s arm postures on users’ perceived safety. A polite, butler-like posture is recommended as the default position, aligning with the role users typically attribute to the robot GARMI.

Keywords: Social HRI, Art and Entertainment Robotics, Deep Learning for Visual Perception
Abstract: The deployment of robots as social gamers offer a wide spectrum to test the usability of complex behaviors. Social robots need to commit with social rules and show gaze related behaviors in order to facilitate communication and reciprocity. In this vein, joint attention is basic to engage users to interact with robots. This paper describes a visual focus of attention extraction module that is further on used to play “I see with my little eye” with Pepper. The degree of sociability of the robot is incremented with head motion that gives the illusion of joint attention, together with deception production and detection capabilities, body expression and some degree of initiative to start the game. The robot global ability is evaluated by a population of N=15 participants that reveal that the main discomfort is generated not by the lack of precision of the gaze estimation module. This dis-function is assumed as part of the game, the robot is allowed to make mistakes, contrary to dialogue manager system that guides the game. Not being understood by the robot generates frustration.

Keywords: Factory Automation, Perception for Grasping and Manipulation, Grasping
Abstract: Dexterous grasping of unseen objects in dynamic environments is an essential prerequisite for the advanced manipulation of autonomous robots. Prior advances rely on several assumptions that simplify the setup, including environment stationarity, pre-defined objects, and low-dimensional end-effectors. Though easing the problem and enabling progress, it undermined the complexity of the real world. Aiming to relax these assumptions, we present a dynamic grasping framework for unknown objects in this work, which uses a five-fingered hand with visual servo control and can compensate for external disturbances. To establish such a system on real hardware, we leverage the recent advances in real-time dexterous generative grasp synthesis and introduce several techniques to secure the robustness and performance of the overall system. Our experiments on real hardware verify the ability of the proposed system to reliably grasp unknown dynamic objects in two realistic scenarios: objects on a conveyor belt and human-robot handover. Note that there has been no prior work that can achieve dynamic multi-fingered grasping for unknown objects like ours up to the time of writing this paper. We hope our pioneering work in this direction can provide inspiration to the community and pave the way for further algorithmic and engineering advances on this challenging task. A video of the experiments is available at href{https://youtu.be/Pw9jEV6FUfE}.

Keywords: Perception for Grasping and Manipulation, Deep Learning for Visual Perception, Data Sets for Robotic Vision
Abstract: This paper studies a new open-set problem, the open-vocabulary category-level object pose and size estimation. Given human text descriptions of arbitrary novel object categories, the robot agent seeks to predict the position, orientation, and size of the target object in the observed scene image. To enable such generalizability, we

first introduce OO3D-9D, a large-scale photorealistic dataset for this task. Derived from OmniObject3D, OO3D-9D is the largest and most diverse dataset in the field of category-level object pose and size estimation. It includes additional annotations for the symmetry axis of

each category, which help resolve symmetric ambiguity. Apart from the large-scale dataset, we find another key factor to enabling such generalizability is leveraging the strong prior knowledge

in pre-trained visual-language foundation models. We then propose a framework built on pre-trained DinoV2 and text-to-image stable diffusion models to infer the normalized object coordinate space (NOCS) maps of the target instances. This framework fully leverages the visual semantic prior from DinoV2 and the aligned visual and language knowledge within the text-to-image diffusion model,

which enables generalization to various text descriptions of novel categories. Comprehensive quantitative and qualitative experiments demonstrate that

the proposed open-vocabulary method, trained on our large-scale synthesized data, significantly outperforms the baseline and can effectively generalize to real-world images of unseen categories. The code and data of our method will be made public.

Keywords: Human-Robot Collaboration, Acceptability and Trust, Human-Robot Teaming
Abstract: Recent research showed that enhancing robots with human-like appearance and movements can significantly improve human-robot collaboration. These improvements are mainly seen in increased trust and a better understanding of the mechanical system.

In this work, we investigated the role of robot anthropomorphism in teleoperation contexts, demonstrating that it positively impacts both the users' experience and their performance. More specifically, we focused on analyzing the contributions of anthropomorphism in reaching and grasping tasks within a virtual environment. For each task, subjects were asked to control avatars with different anthropomorphism scores. Performance and users' feelings were collected and compared by means of a statistical analysis. All participants followed shorter trajectories in less time when controlling a human-like avatar, with mean reductions of 11.56s and 0.44 m compared to controlling a robot-like avatar in the best-case scenario. Similarly, grasping abilities were superior when using a more anthropomorphic end-effector with respect to controlling grippers with two or three fingers.

Keywords: Software-Hardware Integration for Robot Systems, Humanoid Robot Systems, Dual Arm Manipulation
Abstract: Flexible object manipulation of paper and cloth is a ma- jor research challenge in robot manipulation. Although there have been efforts to develop hardware that enables specific actions and to realize a single action of paper folding using sim-to-real and learning, there have been few proposals for humanoid robots and systems that enable continuous, multi- step actions of flexible materials. Wrapping an object with paper and tape is more complex and diverse than traditional manipulation research due to the increased number of objects that need to be handled, as well as the three-dimensionality of the operation. In this research, necessary information is organized and coded based on the characteristics of each object handled in wrapping. We also generalize the hardware configuration, manipulation method, and recognition system that enable humanoid wrapping operations. The system will include manipulation with admittance control focusing on paper tension and state evaluation using point clouds to handle three- dimensional flexible objects. Finally, wrapping objects with different shapes is experimented with to show the generality and effectiveness of the proposed system.

Keywords: Product Design, Development and Prototyping, Grippers and Other End-Effectors, Additive Manufacturing
Abstract: As a problem-solving activity, engineering design is usually iterative involving multiple proposed solutions that are tested against a predefined set of constraints. Human designers usually rely on their knowledge, experience, and intuition, which is a drawback when dealing with certain unknown problems. This is easily overcome by an AI that can generate and test several thousand alternative solutions to a design problem iteratively in the form of a parametric computational model. This paper seeks to present one such automated design process involving the development and testing of a low-maintenance robotic gripper featuring underactuation and reduced weight for missions in extreme environments. This is achieved by considering the computer as a collaborative partner in the design process, where the cloud computing engines generate thousands of mechanically improved designs in response to our rigorous and robust input computational model. Generated solutions include uniquely synthesised structures designed to achieve the aforementioned objectives. Notable contributions of this paper are presented through a comparative study confirming the gripper's improved component accessibility, structural resilience, and doubled weight-to-power ratio achieved through 73% crude weight reduction compared to its predecessor.

Keywords: Body Balancing, Humanoid and Bipedal Locomotion, Whole-Body Motion Planning and Control
Abstract: A novel slide–steer gait is proposed to enable a humanoid to adjust its orientation in extreme environments. The proposed gait uses the thrust of a ducted fan to make the robot steer a large angle, even when the robot is standing on only one leg. The kinetics model of the gait is carefully analyzed, and a corresponding planning method considering energy saving is developed. In addition, a control strategy is proposed to enable the robot to resist ground friction and ensure the stability and accuracy of tracking the planned trajectory of the swinging foot. Finally, the method is examined on a prototype robot, Jet-HR3. Using the proposed gait, the robot successfully achieved a large turn (260°) on a corner with a narrow landing area in only 4 s.

Keywords: Multi-Robot Systems, Telerobotics and Teleoperation, Virtual Reality and Interfaces
Abstract: As multitasking situations keep increasing in our daily lives, telepresence robots have been used to embody a human’s consciousness to a remote location with minimal effort. However, the number of robots that one person can manage at once is limited. In these situations, the human’s cognitive puts a limit on how many or how much can one person control such robots remotely. This paper takes an overview on what applications multi-presence could be applied in and present a system design to evaluate the cognitive effect of when users control up to 4 robots at one time with varying settings. This study reveals the cognitive threshold at which managing three robots optimizes task performance while minimizing cognitive load in multi-presence robotic systems.

Keywords: Wearable Robotics, Prosthetics and Exoskeletons, Optimization and Optimal Control
Abstract: This paper presents advancements in the functionalities of the Recupera-Reha lower extremity exoskeleton robot. The exoskeleton features a series-parallel hybrid design characterized by multiple kinematic loops, resulting in 148 degrees of freedom in its spanning tree and 102 independent loop closure constraints, which pose significant challenges for modeling and control. To address these challenges, we applied an optimal control approach to generate feasible trajectories such as sitting, standing, and static walking and tested these trajectories on the exoskeleton robot. Our method efficiently solves the optimal control problem using a serial abstraction of the model to generate trajectories. It then utilizes the full series-parallel hybrid model, which takes all the kinematic loop constraints into account to generate the final actuator commands. The experimental results demonstrate the effectiveness of our approach in generating the desired motions for the exoskeleton.

Keywords: Modeling and Simulating Humans, Bioinspired Robot Learning, Motion Control
Abstract: In recent years, bio-inspired control algorithms for robotic arms have garnered significant attention due to their remarkable adaptability. In this paper, we introduce a real-time framework on the basis of CBMC[13], to control a 7-DOF robotic arm using neuromorphic chip. Our proposed framework comprises five modules: the cerebral sensory cortex module, the cerebral motor cortex module, the cerebellum module, the brainstem module, and the spinal cord module. These modules operate within a hierarchical structure consisting of three control loops, each running at different frequencies. The effectiveness of our approach is validated through trajectory tracking control tasks in simulation. Subsequently, we deploy the algorithm on a neuromorphic chip to test its performance on a robotic arm platform. The experimental results clearly demonstrate the superior control effectiveness and robustness of our methodology than the previous version in complex tasks.

Keywords: Optimization and Optimal Control, Whole-Body Motion Planning and Control
Abstract: Quadratic Programs (QPs) have become a mature technology for the control of robots of all kinds, including humanoid robots. One aspect has been largely overlooked, however, which is the accuracy with which these QPs should be solved. QP solvers aim at providing solutions accurate up to floating point precision (approx10^{-8}). Considering physical quantities expressed in SI or similar units (meters, radians, etc.), such precision seems completely unrelated to both task requirements and hardware capacity. Typically, humanoid robots never achieve, nor are capable of achieving sub-millimeter precision in manipulation tasks. With this observation in mind, our objectives in this paper are two-fold: first examine how the QP solution accuracy impacts the resulting robot motion accuracy, then evaluate how a reduced solution accuracy requirement can be leveraged to reduce the corresponding computational effort. Experiments with a dynamic simulation of RHPS-1 humanoid robot indicate that computational effort can be divided by more than 27 while maintaining the desired motion accuracy.

Keywords: Localization, Mapping
Abstract: In this paper, we propose a WiFi-Visual robot localization method for limiting the unbounded error of image-only localization due to visual environment similarity. The localization problem is modeled as a classification problem based on the WiFi-Visual data collected at labelled positions. The heterogeneous WiFi-Visual data are harmonized by representing the WiFi features in image form to adapt to the strong image processing capacity of the neural network. The WiFi features in image form are fused with the visual features provided by the robot camera. The fused WiFi-Visual features are jointly exploited by a neural network to classify WiFi-Visual features of an unknown position to the most likely class. The labelled position corresponding to the most likely class is taken as the estimated position of the robot. Experiments are carried out on the physical robot platform TIAGO++, which can provide the real-time ground truth reference position. Experiment results show that the proposed WiFi-Visual data fusion method can effectively limit the exceptional unbounded localization errors of image-only localization. The RMSE of the proposed method is less than 2 meters. This value is smaller than that of WiFi localization. The proposed method has more stable performance than WiFi-only localization and image-only localization. Its performance can be further improved by Kalman filtering. The demo video is also provided.

Keywords: Computer Vision for Automation, Deep Learning for Visual Perception, Learning from Demonstration
Abstract: Humanoid robots can benefit from their similarity to the human shape by learning from humans. When humans teach other humans how to perform actions, they often demonstrate the actions, and the learning human imitates the demonstration to get an idea of how to perform the action. Being able to mentally transfer from a demonstration seen from a third-person perspective to how it should look from a first-person perspective is fundamental for this ability in humans. As this is a challenging task, it is often simplified for robots by creating demonstrations from the first-person perspective. Creating these demonstrations allows for an easier imitation but requires more effort. Therefore, we introduce a novel diffusion model that enables the robot to learn from the third-person demonstrations directly by learning to generate the first-person perspective from the third-person perspective. The model translates the size and rotations of objects and the environment between the two perspectives. This allows us to utilise the benefits of easy-to-produce third-person demonstrations and easy-to-imitate first-person demonstrations. Our approach significantly outperforms other image-to-image models in this task.

Keywords: Grippers and Other End-Effectors, Perception for Grasping and Manipulation, Hardware-Software Integration in Robotics
Abstract: Modern humanoid robots have shown their promising potential for executing various tasks involving the grasping and manipulation of objects using their end-effectors. Nevertheless, in the most of the cases, the grasping and manipulation actions involve low to moderate payload and interaction forces. This is due to limitations often presented by the end-effectors, which can not match their arm-reachable payload, and hence limit the payload that can be grasped and manipulated. In addition, grippers usually do not embed adequate perception in their hardware, and grasping actions are mainly driven by perception sensors installed in the rest of the robot body, frequently affected by occlusions due to the arm motions during the execution of the grasping and manipulation tasks. To address the above, we developed a modular high grasping force gripper equipped with embedded multi-modal perception functionalities. The proposed gripper can generate a grasping force of 110 N in a compact implementation. The high grasping force capability is combined with embedded multi-modal sensing, which includes an eye-in-hand camera, a Time-of-Flight (ToF) distance sensor, an Inertial Measurement Unit (IMU) and an omnidirectional microphone, permitting the implementation of perception-driven grasping functionalities.

We extensively evaluated the grasping force capacity of the gripper by introducing novel payload evaluation metrics that are a function of the robot arm's dynamic motion and gripper thermal states. We also evaluated the embedded multi-modal sensing by performing perception-guided enhanced grasping operations.

Keywords: Collision Avoidance, Mobile Manipulation, Reinforcement Learning
Abstract: Pedipulation leverages the feet of legged robots for mobile manipulation, eliminating the need for dedicated robotic arms. While previous works have showcased blind and task-specific pedipulation skills, they fail to account for static and dynamic obstacles in the environment. To address this limitation, we introduce a reinforcement learning-based approach to train a whole-body obstacle-aware policy that tracks foot position commands while simultaneously avoiding obstacles. Despite training the policy in only five different static scenarios in simulation, we show that it generalizes to unknown environments with different numbers and types of obstacles. We analyze the performance of our method through a set of simulation experiments and successfully deploy the learned policy on the ANYmal quadruped, demonstrating its capability to follow foot commands while navigating around static and dynamic obstacles.

Keywords: Humanoid and Bipedal Locomotion, Legged Robots, Reinforcement Learning
Abstract: This paper introduces a new approach to enhance the robustness of humanoid walking under strong perturbations, such as substantial pushes. Effective recovery from external disturbances requires bipedal robots to dynamically adjust their stepping strategies, including footstep positions and timing. Unlike most advanced walking controllers that restrict footstep locations to a predefined convex region, substantially limiting recoverable disturbances, our method leverages reinforcement learning to dynamically adjust the permissible footstep region, expanding it to a larger, effectively non-convex area and allowing cross-over stepping, which is crucial for counteracting large lateral pushes. Additionally, our method adapts footstep timing in real-time to further extend the range of recoverable disturbances. Based on the adjustments, feasible footstep positions and DCM trajectory are planned by solving a QP. Finally, we employ a DCM controller together with an inverse dynamics whole-body control framework to ensure the robot effectively follows the trajectory.

Keywords: Human-Robot Collaboration, Physical Human-Robot Interaction, Haptics and Haptic Interfaces
Abstract: Object handovers, a seemingly straightforward action, involve a complex interplay of predictive and reactive control mechanisms in both partners. Understanding the cues that are used by humans to predict object properties is needed for planning natural robot handovers. In human-human interactions, the receiver can extract information from the passer's movement. Here, we show in a VR simulated agent-human object handover, that the human receiver can use passer kinematic cues to predict the transported object's properties, such as weight, and preemptively adapt the grasping strategy towards them. We show that when the agent's movement is correlated to the object weight, humans can interpret this cue and produce proportional anticipatory grip forces before object release. This adaptation is learned even when objects are presented in a random order and is strengthened with the repeated presentation of the pairing. The outcome of this study contributes to a better understanding of non-verbal cues in handover tasks and enables more transparent and efficient real-world physical robot-human interactions.

Keywords: Deep Learning in Grasping and Manipulation, Imitation Learning, Dexterous Manipulation
Abstract: In the context of imitation learning applied to anthropomorphic robotic hands, the high complexity of the systems makes learning complex manipulation tasks challeng- ing. However, the numerous datasets depicting human hands in various different tasks could provide us with better knowledge regarding human hand motion. We propose a method to leverage multiple large-scale task-agnostic datasets to obtain latent representations that effectively encode motion subtra- jectories that we included in a transformer-based behavior cloning method. Our results demonstrate that employing la- tent representations yields enhanced performance compared to conventional behavior cloning methods, particularly regarding resilience to errors and noise in perception and proprioception. Furthermore, the proposed approach solely relies on human demonstrations, eliminating the need for teleoperation and, therefore, accelerating the data acquisition process. Accurate inverse kinematics for fingertip retargeting ensures precise transfer from human hand data to the robot, facilitating effective learning and deployment of manipulation policies. Finally, the trained policies have been successfully transferred to a real-world 23Dof robotic system.

Keywords: Telerobotics and Teleoperation, Humanoid Robot Systems, Human-Robot Teaming
Abstract: Teleoperation of humanoid robots has long been a challenging domain, necessitating advances in both hardware and software to achieve seamless and intuitive control. This paper presents an integrated solution based on several elements: calibration-free motion capture and retargeting, low-latency fast whole-body kinematics streaming toolbox and high-bandwidth cycloidal actuators. Our motion retargeting approach stands out for its simplicity, requiring only 7 IMUs to generate full-body references for the robot. The kinematics streaming toolbox, ensures real-time, responsive control of the robot's movements, significantly reducing latency and enhancing operational efficiency. Additionally, the use of cycloidal actuators makes it possible to withstand high speeds and impacts with the environment. Together, these approaches contribute to a teleoperation framework that offers unprecedented performance. Experimental results on the humanoid robot Nadia demonstrate the effectiveness of the integrated system.

Keywords: Dynamics, Simulation and Animation, Legged Robots
Abstract: Designs incorporating kinematic loops are becoming increasingly prevalent in the robotics community. Despite the existence of dynamics algorithms to deal with the effects of such loops, many modern simulators rely on dynamics libraries that require robots to be represented as kinematic trees. This requirement is reflected in the de facto standard format for describing robots, the Universal Robot Description Format (URDF), which does not support kinematic loops resulting in closed chains. This paper introduces an enhanced URDF, termed URDF+, which addresses this key shortcoming of URDF while retaining the intuitive design philosophy and low barrier to entry that the robotics community values. The URDF+ keeps the elements used by URDF to describe open chains and incorporates new elements to encode loop joints. We also offer an accompanying parser that processes the system models coming from URDF+ so that they can be used with recursive rigid-body dynamics algorithms for closed-chain systems that group bodies into local, decoupled loops. This parsing process is fully automated, ensuring optimal grouping of constrained bodies without requiring manual specification from the user. We aim to advance the robotics community towards this elegant solution by developing efficient and easy-to-use software tools.

Keywords: Reinforcement Learning, Humanoid and Bipedal Locomotion, Legged Robots
Abstract: Learning a locomotion controller for a musculoskeletal system is challenging due to over-actuation and high-dimensional action space. While many reinforcement learning methods attempt to address this issue, they often struggle to learn human-like gaits because of the complexity involved in engineering an effective reward function. In this paper, we demonstrate that adversarial imitation learning can address this issue by analyzing key problems and providing solutions using both current literature and novel techniques. We validate our methodology by learning walking and running gaits on a simulated humanoid model with 16 degrees of freedom and 92 Muscle-Tendon Units, achieving natural-looking gaits with only a few demonstrations.

Keywords: Legged Robots, Sensor Fusion, Localization
Abstract: Accurate state estimation is crucial for legged robot locomotion, as it provides the necessary information to allow control and navigation. However, it is also challenging, especially in scenarios with uneven and slippery terrain. This paper presents a new Invariant Extended Kalman filter for legged robot state estimation using only proprioceptive sensors. We formulate the methodology by combining recent advances in state estimation theory with the use of robust cost functions in the measurement update. We tested our methodology on quadruped robots through experiments and public datasets, showing that we can obtain a pose drift up to 40% lower in trajectories covering a distance of over 450m, in comparison with a state-of-the-art Invariant Extended Kalman filter.

Keywords: Art and Entertainment Robotics, In-Hand Manipulation, Reinforcement Learning
Abstract: Audio information plays an important role in various robotic manipulation tasks, such as pouring and music performance, as the produced audio can serve as an informative indicator for evaluating actions. However, it is rarely explored in reinforcement learning methods. Due to the unique nature of audio information, it is challenging to simulate in a simulator or use it as direct feedback. Therefore, in this paper, we propose a reinforcement learning method based on audio feedback, aiming to train a dexterous hand to play the xylophone in the real world. By optimizing the dexterous hand's actions using the produced audio, we can make the characteristics of the audio—such as amplitude, waveform shape, and timing—similar to human performance.

Keywords: Humanoid and Bipedal Locomotion, Hardware-Software Integration in Robotics, Legged Robots
Abstract: In this paper, we present a new, standalone bipedal robot system that features Hybrid Leg linkages and a 2-DOF neck to incorporate a camera. An online walking pattern generator using ZMP preview control enables remote robot operation based on user commands. The foot trajectory is modified by IMU sensor feedback for stabilization. Taking advantage of the low structural inertia of Hybrid Leg, the effectiveness of bipedal locomotion with pattern generation based on the cart-table model is tested. Given the bio-inspired nature of the Hybrid Leg structure, the gait pattern is chosen to emulate that of humans. We demonstrate bipedal locomotion on an approximately flat table while controlling it remotely. Simultaneous control of two Hybrid Leg bipedal robots is also shown to highlight the performance of the system.

Keywords: Learning from Demonstration, Bimanual Manipulation, Humanoid Robot Systems
Abstract: In this paper, we present a novel approach for learning bimanual manipulation actions from human demonstration by extracting spatial constraints between affordance regions, termed affordance constraints, of the objects involved. Affordance regions are defined as object parts that provide interaction possibilities to an agent. For example, the bottom of a bottle affords the object to be placed on a surface, while its spout affords the contained liquid to be poured. We propose a novel approach to learn changes of affordance constraints in human demonstration to construct spatial bimanual action models representing object interactions. To exploit the information encoded in these spatial bimanual action models, we formulate an optimization problem to determine optimal object configurations across multiple execution keypoints while taking into account the initial scene, the learned affordance constraints, and the robot’s kinematics. We evaluate the approach in simulation with two example tasks (pouring drinks and rolling dough) and compare three different definitions of affordance constraints: (i) component-wise distances between affordance regions in Cartesian space, (ii) component-wise distances between affordance regions in cylindrical space, and (iii) degrees of satisfaction of manually defined symbolic spatial affordance constraints.

Keywords: Prosthetics and Exoskeletons, Human Factors and Human-in-the-Loop, Virtual Reality and Interfaces
Abstract: Movement-based strategies are being explored as alternatives to unsatisfactory myoelectric controls for transhumeral prostheses. We recently showed that adding movement goals to shoulder information enabled Artificial Neural Networks (ANNs), trained on natural arm movements, to predict distal joints so well that transhumeral amputees could reach as with their valid arm in Virtual Reality (VR). This control relies on the object’s pose in a shoulder-centered reference frame, whereas it might only be available in a head-centered reference frame through gaze-guided computer vision. Here, we designed two methods to perform the required head-to-shoulder transformation from orientation-only data, possibly available in real-life settings. The first involved an ANN trained offline to do this transformation, while the second was based on a bioinspired space map with online adaptation. Experimental results on twelve participants controlling a prosthesis avatar in VR demonstrated persistent errors with the first method, while the second method effectively encoded the transition between the two frames. The effectiveness of this second method was also tested on six transhumeral amputees in VR, and a physical proof of concept was implemented on a teleoperated robotic platform with computer vision. Those advances represent necessary steps toward the deployment of movement-based control in real-life scenarios.

Keywords: Dexterous Manipulation, Deep Learning in Grasping and Manipulation, Multifingered Hands
Abstract: This work focuses on the problem of robotic picking in challenging multi-object scenarios. These scenarios include difficult-to-pick objects (e.g. too small, too flat objects) and challenging conditions (e.g. objects obstructed by other objects and/or the environment). To solve these challenges, we leverage four dexterous picking skills inspired by human manipulation techniques and propose methods based on deep neural networks that predict when and how to apply the skills based on the shape of the objects, their relative locations to each other, and the environmental factors. We utilize a compliant, under-actuated hand to reliably apply the identified skills in an open-loop manner. The capabilities of the proposed system are evaluated through a series of real-world experiments, comprising 45 trials with 150+ grasps, to assess its reliability and robustness, particularly in cluttered settings. This research helps bridge the gap between human and robotic grasping, showcasing promising results in various practical scenarios.

Keywords: Imitation Learning, Legged Robots, Intention Recognition
Abstract: As humanoid robots transition from labs to real-world environments, it is essential to democratize robot control for non-expert users. Recent human-robot imitation algorithms focus on following a reference human motion with high precision, but they are susceptible to the quality of the reference motion and require the human operator to simplify its movements to match the robot's capabilities. Instead, we consider that the robot should understand and adapt the reference motion to its own abilities, facilitating the operator's task. For that, we introduce a deep-learning model that anticipates the robot's performance when imitating a given reference. Then, our system can generate multiple references given a high-level task command, assign a score to each of them, and select the best reference to achieve the desired robot behavior. Our Self-AWare model (SAW) ranks potential robot behaviors based on various criteria, such as fall likelihood, adherence to the reference motion, and smoothness. We integrate advanced motion generation, robot control, and SAW in one unique system, ensuring optimal robot behavior for any task command. For instance, SAW can anticipate falls with 99.29% accuracy.

Keywords: Reinforcement Learning, Dual Arm Manipulation
Abstract: Curriculum reinforcement learning (CRL) enables learning optimal policies in complex tasks such as robotic hand manipulation. However, in tasks with high-dimensional contexts with temporally continuous goals, previous research has encountered issues such as increased computational costs and the inability to create appropriate curricula to facilitate learning. Therefore, this study proposes a novel CRL method that appropriately segments high-dimensional contexts and learns them using a generative model. Additionally, we propose a method to further enhance learning by incorporating difficulty information into the generative model. Finally, we experimentally confirm that our proposed method significantly accelerates learning in complex tasks such as dual-arm dexterous hand tasks, specifically, RoboPianist.

Keywords: Humanoid Robot Systems, Tendon/Wire Mechanism, Biomimetics
Abstract: Humanoids exhibit a wide variety in terms of joint configuration, actuators, and degrees of freedom, resulting in different achievable movements and tasks for each type. Particularly, musculoskeletal humanoids are developed to closely emulate human body structure and movement functions, consisting of a skeletal framework driven by numerous muscle actuators. The redundant arrangement of muscles relative to the skeletal degrees of freedom has been used to represent the flexible and complex body movements observed in humans. However, due to this flexible body and high degrees of freedom, modeling, simulation, and control become extremely challenging, limiting the feasible movements and tasks. In this study, we integrate the musculoskeletal humanoid Musashi with the wire-driven robot CubiX, capable of connecting to the environment, to form CubiXMusashi. This combination addresses the shortcomings of traditional musculoskeletal humanoids and enables movements beyond the capabilities of other humanoids. CubiXMusashi connects to the environment with wires and drives by winding them, successfully achieving movements such as pull-up, rising from a lying pose, and mid-air kicking, which are difficult for Musashi alone. This concept demonstrates that various humanoids, not limited to musculoskeletal humanoids, can mitigate their physical constraints and acquire new abilities by connecting to the environment and driving through wires.

Keywords: Telerobotics and Teleoperation, Humanoid Robot Systems, Virtual Reality and Interfaces
Abstract: Teleoperation plays a crucial role in enabling robot operations in challenging environments, yet existing limitations in effectiveness and accuracy necessitate the development of innovative strategies for improving teleoperated tasks. This article introduces a novel approach that utilizes mixed reality and assistive autonomy to enhance the efficiency and precision of humanoid robot teleoperation. By leveraging Probabilistic Movement Primitives, object detection, and Affordance Templates, the assistance combines user motion with autonomous capabilities, achieving task efficiency while maintaining human-like robot motion. Experiments and feasibility studies on the Nadia robot confirm the effectiveness of the proposed framework.

Keywords: Human and Humanoid Motion Analysis and Synthesis, Modeling and Simulating Humans, Datasets for Human Motion
Abstract: This paper proposes a framework for measuring human motions involving surface contacts by collecting data from distributed tactile sensors and motion capture systems simultaneously. Although contacts play an important role in natural robot interaction with humans and environments, their high complexity makes contact-rich motions challenging for even advanced humanoid robots. One possible approach is to learn from humans who generate such motions with ease in their daily lives. While analysis of human contact motions can lead to understanding human motion strategy to improve robots' motion capacity and robustness, access to human motion data including contacts is still limited. This paper addresses this issue by establishing a method for obtaining human motions with wide-area contacts. The contact information measured by the tactile sensors is mapped on the human body through position-orientation and force registration, and unified with synchronized body motion data. A series of experiments have been conducted to validate the physical quality of the force measurement and demonstrate that the proposed framework is effective in acquiring whole-body contact motions.

Keywords: Human Detection and Tracking, Perception-Action Coupling, Human-Robot Collaboration
Abstract: Gaze behaviour plays a crucial role in social interactions extending to Human-Robot Interactions. Humanoid robots equipped with anthropomorphic vision systems may be expected to employ natural gaze behaviour in interactions with humans. Here, we implemented a novel first-person human gaze state detection tool on the iCub robot and assessed its gaze detection accuracy. Using this tool, we then developed two social gaze interaction architectures for the iCub robot. The Social Gaze Space theory (SGS) of Jording and colleagues was the theoretical framework for building our first architecture, SGS Base (SGS-B). Our second architecture, SGS Interaction Architecture (SGS-IA), is the extended version of SGS Base that lets the robot take on different higher-level goals in an interaction, including being attentive to the interaction partner or attempting to direct their attention to an object. These architectures support dyadic social interactions involving initiating and responding to joint attention towards objects of interest, and are expected to allow the robot to interact in a more human-like manner in social interactions. The gaze detection accuracy of the iCub robot equipped with our choice of algorithms exceeded that of previously implemented methods. System validation trials confirmed that the robot detects the social gaze states correctly in the majority of the instances, allowing for behavioural control according to the Social Gaze Space theory. Validation tests presented in this article demonstrate the functionality of the SGS Interaction Architecture and also highlight the differences in personal interaction dynamics when the humanoid robot has no particular goals as opposed to when it has interactional goals appropriate to a teaching scenario.

Keywords: Social HRI, Human-Centered Robotics, Rehabilitation Robotics
Abstract: Joint Attention (JA) is a fundamental social interaction capability developed in early childhood by sharing a common focus point with others. However, this skill is commonly affected in children with Autism Spectrum Disorders (ASD). In this sense, social robots emerged as a tool for developing novel JA intervention strategies supporting therapists. Therefore, this work presents a study to determine the best combination of robot and therapist participation in JA therapies based on following the gaze of the mediator with verbal and pointing instructions. Sixteen subjects with ASD participated in seven robot-assisted sessions divided into a robot-assisted group (RAG) and a control group (CG), performing equivalent intervention sessions. A focus visual system measured quantitative and reliable JA metrics, while the robot's participation was progressively increased in the RAG to manage all the instructions at the last session. Results show that participants of the RAG had better JA scores than the CG during sessions 6 and 7 (p = 0.029 and p = 0.018, respectively). The RAG paid more visual attention and presented more social engagement behaviours in the sessions (p < 0.05). Moreover, the best JA performance was given when the therapist and the robot requested similar instructions during the activities.

Keywords: Task Planning, Service Robotics, Cognitive Control Architectures
Abstract: Recent advances in Large Language Models (LLMs) have been instrumental in autonomous robot control and human-robot interaction by leveraging their vast general knowledge and capabilities to understand and reason across a wide range of tasks and scenarios. Previous works have investigated various prompt engineering techniques for improving the performance of LLM to accomplish tasks, while others have proposed methods that utilize LLMs to plan and execute tasks based on the available functionalities of a given robot platform. In this work, we consider both lines of work by comparing prompt engineering techniques and combinations thereof within the application of high-level task planning and execution in service robotics. We define a diverse set of tasks and a simple set of functionalities in simulation, and measure task completion accuracy and time for several state-of-the-art models. We make our code, including all prompts, available at https://github.com/AIS-Bonn/Prompt_Engineering.

Keywords: Rehabilitation Robotics
Abstract: Sarcopenia, the decline in skeletal muscle tone, mass, and strength associated with aging and lack of physical activity, is becoming more common due to the increasing number and proportion of older adults in the global population. Regular physical exercises that target both the upper and lower limbs are essential to prevent the onset of Sarcopenia and counteract its effects. However, motivating older adults to start a fitness routine, especially in a group setting, is a significant challenge. This underscores the need for innovative, engaging solutions that are easy to use and designed for group participation. The main goal of this study was to develop and evaluate a new method to meet this need. We created a platform featuring the humanoid robot Pepper, which led a group of participants through a series of physical exercises designed to prevent and reduce Sarcopenia. The robot demonstrated and performed the exercises alongside the participants. Additionally, an external camera allowed Pepper to monitor the exercises in real-time, encouraging participants who slowed down or did not complete all movements. Offline processing of the recorded data enabled the assessment of individual performance. The platform was tested with 8 subjects diagnosed with Sarcopenia, divided into two groups. Preliminary results were promising participants expressed high satisfaction with the robot-guided training. They moved almost synchronously, indicating that they followed the robot’s instructions closely, remained engaged, and adhered to the correct timing of the exercises.

Keywords: Learning from Demonstration, Deep Learning Methods, Motion Control
Abstract: We present a novel motion generation approach for robot arms, with high degrees of freedom, in complex settings that can adapt online to obstacles or new via points. Learning from Demonstration facilitates rapid adaptation to new tasks and optimizes the utilization of accumulated expertise by allowing robots to learn and generalize from demonstrated trajectories. We train a transformer architecture, based on conditional variational autoencoder, on a large dataset of simulated trajectories used as demonstrations. Our architecture learns essential motion generation skills from these demonstrations and is able to adapt them to meet auxiliary tasks. Additionally, our approach implements auto-regressive motion generation to enable real-time adaptations, as, for example, introducing or changing via-points, and velocity and acceleration constraints. Using beam search, we present a method for further adaption of our motion generator to avoid obstacles. We show that our model successfully generates motion from different initial and target points and that is capable of generating trajectories that navigate complex tasks across different robotic platforms.

Keywords: AI-Enabled Robotics, Dual Arm Manipulation
Abstract: Although there has been rapid progress in endowing robots with the ability to solve complex manipulation tasks, generating control policies for bimanual robots to solve tasks involving two hands is still challenging because of the difficulties in effective temporal and spatial coordination. With emergent abilities in terms of step-by-step reasoning and in-context learning, Large Language Models (LLMs) have demonstrated promising potential in a variety of robotic tasks. However, the nature of language communication via a single sequence of discrete symbols makes LLM-based coordination in continuous space a particular challenge for bimanual tasks. To tackle this challenge, we present LAnguage-model-based Bimanual ORchestration (LABOR), an agent utilizing an LLM to analyze task configurations and devise coordination control policies for addressing long-horizon bimanual tasks. We evaluate our method through simulated experiments involving two classes of long-horizon tasks using the NICOL humanoid robot. Our results demonstrate that our method outperforms the baseline in terms of success rate. Additionally, we thoroughly analyze failure cases, offering insights into LLM-based approaches in bimanual robotic control and revealing future research trends. The project website can be found at http://labor-agent.github.io.

Keywords: Multi-Contact Whole-Body Motion Planning and Control, Humanoid and Bipedal Locomotion, Optimization and Optimal Control
Abstract: Model predictive control is one of the most common methods for stabilizing the dynamics of a legged robot. Yet, it remains unclear which level of complexity should be considered for modeling the system dynamics. On the one hand, most embedded pipelines for legged locomotion rely on reduced models with low computational load in order to ensure real-time capabilities at the price of not exploiting the full potential of the whole-body dynamics. On the other hand, recent numerical solvers can now generate whole-body trajectories on the fly while still respecting tight time constraints. This paper compares the performances of common dynamic models of increasing complexity (centroidal, kino-dynamics, and whole-body models) in simulation over locomotion problems involving challenging gaits, stairs climbing and balance recovery. We also present a 3-D kino-dynamics model that reformulates centroidal dynamics in the coordinates of the base frame by efficiently leveraging the centroidal momentum equation at the acceleration level. This comparative study uses the humanoid robot Talos and the augmented Lagrangian-based solver ALIGATOR to enforce hard constraints on the optimization problem.

Keywords: Compliance and Impedance Control, Machine Learning for Robot Control, Dual Arm Manipulation
Abstract: Variable Impedance Control (VIC) approaches offer effective means for enabling robots to perform physical interaction tasks safely and proficiently, by including time-varying gains within an impedance control loop. However, determining the optimal gain profiles can be tedious and time-consuming. To address this challenge, this study introduces a VIC learning framework capable of autonomously acquiring suitable impedance behavior during task execution. This achievement is realized through the fusion of two techniques: (i) Reinforcement Learning (RL), to determine the most appropriate stiffness and damping gains for solving interaction tasks (e.g., lifting, pushing); and (ii) Gaussian Processes (GPs) for modeling and estimating optimal impedance parameters across task variations (e.g., changes in object weight). Consequently, we propose a Fast Cross-Entropy Method (FCEM) algorithm for autonomous stiffness learning, emphasizing all-the-time-stability to guarantee the stability of the control loop throughout the RL process. Additionally, we present a GP-based method to adapt impedance behaviors at run-time, adjusting stiffness based on online external torques estimates provided by a momentum observer (without requiring a wrench sensor). Experimental results on a simulated ABB Mobile YuMi robot show the framework’s capabilities across different tasks.

Keywords: Telerobotics and Teleoperation, Virtual Reality and Interfaces, Human-Centered Automation
Abstract: Robotic arms demonstrate superior speed and precision when performing tasks in challenging environments. However, creating fully autonomous robots remains challenging, particularly for tasks that demand the kind of subtle perception and complex decision-making that humans use naturally. This paper investigates the integration of Virtual Reality (VR) with robotic telemanipulation, aiming to boost cooperation between humans and robots in the execution of peg-in-hole assembly tasks. The introduced semi-autonomous framework utilizes visual object detection technology to identify the types and 6D poses of objects, offering affordances to the user. The robot autonomously plans its movement toward the designated goal location before switching to the real-time telemanipulation scheme. This allows the user to either approve the pre-determined task execution plan or take over the robot control and make necessary adjustments. Following a comparative user study between the proposed framework and a pure telemanipulation system, the effectiveness of this approach is evaluated and demonstrated.

Keywords: Prosthetics and Exoskeletons, Physical Human-Robot Interaction, Physically Assistive Devices
Abstract: Over the past decade, industrial ergonomics have made significant advances, leading to the development of various occupational exoskeletons. While beneficial, exoskeletons could disrupt motor control due to their distributed interaction with the human body. This study explores individual factors influencing different adaptation patterns following exoskeleton use in asymptomatic individuals. Fifty-five participants used a 4 Degree of Freedom (DoF) arm exoskeleton to perform reaching tasks under low-magnitude force fields. Pre- and post-exposure movements were recorded via motion capture, and personal characteristics were documented. Spectral clustering identified variations in inter-joint coordination after exposition to the exoskeleton, and a random forest classifier linked these patterns to individual anthropometric, demographic and kinematic traits. The model highlighted factors such as laterality, forearm length, and some spontaneous kinematics metrics as key predictors of post-adaptation behavior. These findings underscore the need to consider individual profiles to minimize disruptive motor adaptations and improve exoskeleton safe widespread in industrial applications.

Keywords: Grippers and Other End-Effectors, Mechanism Design
Abstract: We present a design optimisation method for the routing of tendons in a tendon-driven mechanism with the objective of maximising force transmission efficiency (FTE). We formulate a friction model for the different routing elements, accounting for routing point radii and slipping/rolling contacts. We then construct a numerical design optimisation problem to optimise the design parameters, routing point locations, for a given tendon routing topology. We apply the method to the design of an existing tendon-driven gripper. The results show that frictional losses can be reduced by approximately half compared to the baseline design, and that taking into account the routing point radii is indeed of significant influence.

Keywords: Deep Learning Methods, Kinematics, Humanoid Robot Systems
Abstract: Solving the Inverse Kinematics Problem is a fundamental skill for humanoid robots and their interaction with the world. In contrast to industrial manipulators, humanoid robots can simultaneously grasp or push objects with two hands. This calls for generating self-collision-free inverse kinematics solutions for both arms in a minimal time. Recent research in the context of single-arm manipulators utilizes deep generative models to obtain a whole set of feasible solutions. This work investigates their performance on the more complex dual-arm problem. For this, we extend the problem space to a dual-arm setup and learn inverse kinematics solutions, providing the two target end effector poses as a conditional variable. We propose an approach based on Conditional Variational Autoencoders and investigate the trade-off between model size, accuracy, and its benefit when being used for seeding a numeric solver. In this context, we also evaluate the influence of flexible learning-based priors against fixed Gaussian priors. Our approach can initialize the solver within 1 ms and drastically increase the number of returned solutions. The results show that even less accurate models can drastically increase the performance of a numeric solver while yielding significantly shorter solving times compared to a state-of-the-art flow-based method.

Keywords: Humanoid and Bipedal Locomotion, Legged Robots
Abstract: We propose an MPC-based decentralized scheme for cooperative transportation between two agents. One agent, the leader, can be either a human or a robot with knowledge of the task. The other, the follower, has no knowledge of the task, and must autonomously decide how to move based on the perceived interaction forces. The robots interact with the object in a compliant way thanks to a hand admittance controller, and the follower continuously adapts its footstep plan in order to accommodate the hand displacement. The combination of these two effects allows the follower to smoothly react to the motion of the leader: it can move omnidirectionally and rotate, as well as accommodate lifting and lowering of the transported object, all while performing obstacle avoidance during footstep placement. We report dynamic simulations on two HRP-4 robots in a number of different scenarios, both when carrying a table and an object with handles.

Keywords: Bimanual Manipulation, Grasping, Datasets for Human Motion
Abstract: Understanding the intricacies in bimanual manipulation tasks is crucial for advancements in robotics and especially humanoid robotics. This exploratory study investigates the correlations between grasp type selection and bimanual coordination patterns, referred to as bimanual categories, in human manipulation tasks. To do so, we use two taxonomies: the Bimanual Manipulation Taxonomy, which defines categories in bimanual manipulation tasks and the GRASP Taxonomy, which defines human grasp types. In our analysis, we use a subset of the Yale Human Grasping Dataset, which includes natural, routine activities of housekeepers. The analysis reveals, amongst others, correlations between Tightly Coupled Symmetric bimanual coordination and Power Grasps. In addition, we identify edge cases such as handling soft and articulated objects, and self-handovers, and provide clear labeling guidelines according to the taxonomy. Soft objects were found to be predominantly handled with Lateral Pinch grasps. This study provides an initial step toward a deeper understanding of the relationship between grasp selection and bimanual coordination.

Keywords: Visual-Inertial SLAM, Collision Avoidance, Motion Control
Abstract: This paper peruses the problem of controlling robotic manipulators by utilizing visual feedback signals while meeting safety requirements and ensuring closed-loop stability. Accordingly, the paper employs a hybrid observer to estimate an end-effector pose (position and orientation) that combines tip-frame acceleration and angular velocity measurements obtained from an inertial measurement unit with bearing measurements of known landmarks obtained from a stereo camera. The estimated pose is then utilized as a Cartesian feedback to close the control loop. A virtual decomposition control (VDC) method is introduced to control the manipulator, as it is well-suited for controlling complex robots like exoskeletons and humanoid robots. For the first time, by utilizing a quadratic program, control barrier functions are incorporated with VDC as a safety filter, enabling the robot manipulator to avoid potential collisions involving both humans and external obstacles and to handle joint limits. By means of the Lyapunov stability theorem, the combined observer-controller scheme is guaranteed to be asymptotically stable. To demonstrate the effectiveness of the proposed vision-based control structure, a simulation study is performed on a 6-degrees-of-freedom (DoF) long-reach manipulator.

Keywords: Education Robotics, Humanoid and Bipedal Locomotion, Robust/Adaptive Control
Abstract: In this paper, we present STRIDE, a Simple, Terrestrial, Reconfigurable, Intelligent, Dynamic, and Educational bipedal platform. STRIDE aims to propel bipedal robotics research and education by providing a cost-effective implementation with step-by-step instructions for building a bipedal robotic platform while providing flexible customizations via a modular and durable design. Moreover, a versatile terrain setup and a quantitative disturbance injection system are augmented to the robot platform to replicate natural terrains and push forces that can be used to evaluate legged locomotion in practical and adversarial scenarios. We demonstrate the functionalities of this platform by realizing an adaptive step-to-step dynamics based walking controller to achieve dynamic walking. Our work with the open-soured implementation shows that STRIDE is a highly versatile and durable platform that can be used in research and education to evaluate locomotion algorithms, mechanical designs, and robust and adaptative controls. Project Repository: https://github.com/well-robotics/STRIDE Project Video: https://youtu.be/wJkxvUG6msU

Keywords: Human-Centered Automation, Human-Centered Robotics, Human-Robot Collaboration
Abstract: Designing an efficient and resilient human-robot collaboration strategy that not only upholds the safety and ergonomics of shared workspace but also enhances the performance and agility of collaborative setup presents significant challenges concerning environment perception and robot control. In this research, we introduce a novel approach for collaborative environment monitoring and robot motion regulation to address this multifaceted problem. Our study proposes novel computation and division of safety monitoring zones, adhering to ISO 13855 and TS 15066 standards, utilizing 2D lasers information. These zones are not only configured in the standard three-layer arrangement but are also expanded into two adjacent quadrants, thereby enhancing system uptime and preventing unnecessary deadlocks. Moreover, we also leverage 3D visual information to track dynamic human articulations and extended intrusions. Drawing upon the fused sensory data from 2D and 3D perceptual spaces, our proposed hierarchical controller stably regulates robot velocity, validated using Lasalle in-variance principle. Empirical evaluations demonstrate that our approach significantly reduces task execution time and system response delay, resulting in improved efficiency and resilience within collaborative settings.

Keywords: Wheeled Robots, Robust/Adaptive Control, Optimization and Optimal Control
Abstract: The Two-Wheeled Inverted Pendulum with Rolljoint (TWIP-R) model and Dynamic Variance Model Predictive Path Integral (DV-MPPI) controller are proposed to simultaneously solve tracking and balancing problems. The TWIP-R model’s additional roll joint allows it to better handle centrifugal forces, offering superior performance in high-speed curved driving compared to the traditional TWIP model. Similar to the TWIP model, the TWIP-R model also cannot achieve position tracking without additional kinematic control due to the linearization process. Therefore, controlling the TWIP-R can be achieved using Model Predictive Path Integral (MPPI), which is capable of handling nonlinear control. However, MPPI often suffers from chattering issues due to the use of Gaussian random noise, leading to control instability. DV-MPPI controller dynamically adjusts random noise variance based on realtime state errors, reducing chattering and improving stability. By controlling the TWIP-R using DV-MPPI, we simultaneously solved the tracking and balancing problems without any additional kinematic tracking control, achieving smooth output. Experimental results show our approach is effective, demonstrating the TWIP-R model’s superior performance in balancing and tracking and the benefits of the DV-MPPI.

Keywords: Humanoid Robot Systems, Actuation and Joint Mechanisms, Kinematics
Abstract: Humanoid robots are expected to interact with humans in built-for-human environments and perform human-like actions. This makes the design and optimization of humanoid robots challenging, in part because of the complexity of human motions. In our previous work, we introduced an approach that automatically calculates the necessary actuator requirements for a given upper-body humanoid robot kinematic performing motions retargeted from human motion data. In this paper, we propose a humanoid robot design framework, which encompasses robot kinematic arrangement selection and actuator optimization based on the actuator requirements data with a focus on the robot upper-body. We also develop a novel actuator optimization index, based on the speed, acceleration, and torque requirements, to help evaluate possible actuator configurations. The potential of the framework is illustrated through a theoretical optimization analysis of the actuator specifications of the humanoid robots armarVI and ARMAR-7, in which the optimal gear ratios of the arm joint actuators are determined based on a novel actuator optimization index for a specific set of human motions.

Keywords: Contact Modeling, Optimization and Optimal Control, Simulation and Animation, Software, Middleware and Programming Environments
Abstract: Physics simulation is ubiquitous in robotics. Whether in model-based approaches (e.g., trajectory optimization), or model-free algorithms (e.g., reinforcement learning), physics simulators are a central component of modern control pipelines in robotics. Over the past decades, several robotic simulators have been developed, each with dedicated contact modeling assumptions and algorithmic solutions. In this article, we survey the main contact models and the associated numerical methods commonly used in robotics for simulating advanced robot motions involving contact interactions. In particular, we recall the physical laws underlying contacts and friction (i.e., Signorini condition, Coulomb’s law, and the maximum dissipation principle), and how they are transcribed in current simulators. For each physics engine, we expose their inherent physical relaxations along with their limitations due to the numerical techniques employed. Based on our study, we propose theoretically grounded quantitative criteria on which we build benchmarks assessing both the physical and computational aspects of simulation. We support our work with an open-source and efficient C++ implementation of the existing algorithmic variations. Our results demonstrate that some approximations or algorithms commonly used in robotics can severely widen the reality gap and impact target applications. We hope this work will help motivate the development of new contact models, contact solvers, and robotic simulators in general, at the root of recent progress in motion generation in robotics.

Keywords: Manipulation Planning, Multi-Contact Whole-Body Motion Planning and Control, Optimization and Optimal Control
Abstract: Humans can exploit contacts anywhere on their body surface to manipulate large and heavy items, objects normally out of reach or multiple objects at once. However, such manipulation through contacts using the whole surface of the body remains extremely challenging to achieve on robots. This can be labelled as Whole-Body Contact-Rich Manipulation (WBCRM) problem. In addition to the high-dimensionality of the Contact-Rich Manipulation problem due to the combinatorics of contact modes, admitting contact creation anywhere on the body surface adds complexity, which hinders planning of manipulation within a reasonable time. We address this computational problem by formulating the contact and motion planning of planar WBCRM as hierarchical continuous optimization problems. To enable this formulation, we propose a novel continuous explicit representation of the robot surface, that we believe to be foundational for future research using continuous optimization for WBCRM. Our results demonstrate a significant improvement of convergence, planning time and feasibility – with, on the average, 99% less iterations and 96% reduction in time to find a solution over considered scenarios, without recourse to prone-to-failure trajectory refinement steps.

Keywords: Imitation Learning, Learning from Demonstration, Data Sets for Robot Learning
Abstract: Safe learning of locomotion skills is still an open problem. Indeed, the intrinsically unstable nature of the open-loop dynamics of locomotion systems renders naive learning from scratch prone to catastrophic failures in the real world. In this work, we investigate the use of iterative algorithms to safely learn locomotion skills from model predictive control (MPC). In our framework, we use MPC as an expert and take inspiration from the safe data aggregation (SafeDAGGER) framework to minimize the number of failures during training of the policy. Through a comparison with other standard approaches such as behavior cloning and vanilla DAGGER, we show that not only our approach has a substantially fewer number of failures during training, but the resulting policy is also more robust to external disturbances.

Keywords: Body Balancing, Legged Robots, Whole-Body Motion Planning and Control
Abstract: This paper presents a novel controller for the bipedal robot Bolt. Our approach leverages a whole-body model predictive controller in conjunction with a footstep sequencer to achieve robust locomotion. Simulation results demonstrate effective velocity tracking as well as push and slippage recovery abilities. In addition to that, we provide a theoretical sensitivity analysis of the footstep sequencing problem to enhance the understanding of the results.

Keywords: Physical Human-Robot Interaction, Safety in HRI, Touch in HRI
Abstract: Artificial electronic skins covering complete robot bodies can make physical human-robot collaboration safe and hence possible. Standards for collaborative robots (e.g., ISO/TS 15066) prescribe permissible forces and pressures during contacts with the human body. These characteristics of the collision depend on the speed of the colliding robot link but also on its effective mass. Thus, to warrant contacts complying with the Power and Force Limiting (PFL) collaborative regime but at the same time maximizing productivity, protective skin thresholds should be set individually for different parts of the robot bodies and dynamically on the run. Here we present and empirically evaluate four scenarios: (a) static and uniform – fixed thresholds for the whole skin, (b) static but different settings for robot body parts, (c) dynamically set based on every link velocity, (d) dynamically set based on effective mass of every robot link. We perform experiments in simulation and on a real 6-axis collaborative robot arm (UR10e) completely covered with sensitive skin (AIRSKIN) comprising eleven individual pads. On a mock pick-and-place scenario with transient collisions with the robot body parts and two collision reactions (stop and avoid), we demonstrate the boost in productivity in going from the most conservative setting of the skin thresholds (a) to the most adaptive setting (d). The threshold settings for every skin pad are adapted with a frequency of 25 Hz. This work can be easily extended for platforms with more degrees of freedom and larger skin coverage (humanoids) and to social human-robot interaction scenarios where contacts with the robot will be used for communication.

Keywords: Human and Humanoid Motion Analysis and Synthesis, Human Detection and Tracking
Abstract: Human motion recognition and motion prediction are essential in human motion analysis. Nowadays, sensors can be seamlessly integrated into clothing using cutting-edge electronic textile (e-textile) technology, allowing long-term recording of human movements outside the laboratory. Motivated by the recent findings that clothing-attached sensors can achieve higher activity recognition accuracy than body-attached sensors, this work investigates the performance of human motion prediction. It reports experiments in which statistical models learnt from the movement of loose clothing are used to predict motion patterns of the body of robotically simulated and real human behaviours. Counter-intuitively, the results show that fabric-attached sensors can have better motion recognition and prediction performance than rigidly-attached sensors. Specifically, The fabric-attached sensor can improve the accuracy up to 40% and requires up to 80% less duration of the past trajectory to achieve high recognition accuracy (i.e., 95%) compared to the rigidly-attached sensor.

Keywords: Humanoid and Bipedal Locomotion, Motion Control, Legged Robots
Abstract: Multi-modal mobility in robots can enable versatile, adaptable, and plastic locomotion in various environments. The additional mode of mobility can allow the robot to perform maneuvers that it can't do with a single mode of locomotion and expand the range of locomotion that the robot can do. In this work, we look at a legged-thruster multi-modal robot, Harpy, to perform a multiple wall jump maneuver and vertically climb inside a vent. The problem is defined using a simplified planar reduced order model using a single inertial body and a hybrid model. The controller utilized MPC while on the wall to satisfy the no-slip ground constraints, and the controller's performance is shown in simulations.

Keywords: Humanoid and Bipedal Locomotion, Legged Robots, Reinforcement Learning
Abstract: For the deployment of legged robots in real-world environments, it is essential to develop robust locomotion control methods for challenging terrains that may exhibit unexpected deformability and irregularity. In this paper, we explore the application of sim-to-real deep reinforcement learning (RL) for the design of bipedal locomotion controllers for humanoid robots on compliant and uneven terrains. Our key contribution is to show that a simple training curriculum for exposing the RL agent to randomized terrains in simulation can achieve robust walking on a real humanoid robot using only proprioceptive feedback. We train an end-to-end bipedal locomotion policy using the proposed approach, and show extensive real-robot demonstration on the HRP-5P humanoid over several difficult terrains inside and outside the lab environment.

Further, we argue that the robustness of a bipedal walking policy can be improved if the robot is allowed to exhibit aperiodic motion with variable stepping frequency. We propose a new control policy to enable modification of the observed clock signal, leading to adaptive gait frequencies depending on the terrain and command velocity. Through simulation experiments, we show the effectiveness of this policy specifically for walking over challenging terrains by controlling swing and stance durations.

The code for training and evaluation is available online: https://github.com/rohanpsingh/LearningHumanoidWalking.

Keywords: Actuation and Joint Mechanisms, Calibration and Identification, Humanoid Robot Systems
Abstract: This paper presents a scalable method for friction identification in robots equipped with electric motors and high-ratio harmonic drives, utilizing Physics-Informed Neural Networks (PINN). This approach eliminates the need for dedicated setups and joint torque sensors by leveraging the robot’s intrinsic model and state data. We present a comprehensive pipeline that includes data acquisition, preprocessing, ground truth generation, and model identification. The effectiveness of the PINN-based friction identification is validated through extensive testing on two different joints of the humanoid robot ergoCub, comparing its performance against traditional static friction models like the Coulomb-viscous and Stribeck-Coulomb-viscous models. Integrating the identified PINN-based friction models into a two-layer torque control architecture enhances real-time friction compensation. The results demonstrate significant improvements in control performance and reductions in energy losses, highlighting the scalability and robustness of the proposed method, also for application across a large number of joints as in the case of humanoid robots.

Keywords: Force and Tactile Sensing, Humanoid and Bipedal Locomotion
Abstract: This paper proposes a tactile sensor array to detect partial footholds and a wrench control method for biped humanoid robots using detected partial footholds.The proposed tactile sensor adopts a high-density FSR (Force Sensitive Resistor) array.It can read data within milliseconds.Filtering methods for its high-speed data which provide accurate estimation of contact areas are also proposed.They enable the computation of optimal contact forces.A wrench control method for biped humanoid robots, based on contact area information from detected partial footholds, enables stable force control even in partial foothold situations.The effectiveness of this advanced control system is evaluated through stabilizing and climbing stepladder experiments on partial footholds using an actual humanoid robot.

Keywords: Reinforcement Learning, In-Hand Manipulation, AI-Based Methods
Abstract: Reinforcement learning (RL) algorithms are traditionally evaluated and compared by their learning trends (i.e., average performance) over trials and time. However, the presence of a single learning trend in a curriculum is, in fact, an assumption. To test this assumption, we used the performance of Proximal Policy Optimization (PPO) under five different curricula aimed at learning dynamic in-hand manipulation tasks. The curricula consisted of different combinations of rewards for lifting and rotating a 5g ball with a three-finger hand with the palm facing down. Mining the performance of all 60 individual trials as time series, we find there are learning trends distinct from the average. We conclude researchers should look beyond the average learning trends when evaluating curriculum learning to fully identify, appreciate, and evaluate the progression of autonomous learning of multi-objective tasks.

Keywords: Imitation Learning, Learning from Demonstration, Multi-Contact Whole-Body Motion Planning and Control
Abstract: Humanoid robots could benefit from using their upper bodies for support contacts, enhancing their workspace, stability, and ability to perform contact-rich and pushing tasks. In this paper, we propose a unified approach that combines an optimization-based multi-contact whole-body controller with Flow Matching, a recently introduced method capable of generating multi-modal trajectory distributions for imitation learning. In simulation, we show that Flow Matching is more appropriate for robotics than Diffusion and traditional behavior cloning. On a real full-size humanoid robot (Talos), we demonstrate that our approach can learn a whole-body non-prehensile box-pushing task and that the robot can close dishwasher drawers by adding contacts with its free hand when needed for balance. We also introduce a shared autonomy mode for assisted teleoperation, providing automatic contact placement for tasks not covered in the demonstrations. Full experimental videos are available at: https://hucebot.github.io/flow_multisupport_website/