Keywords: Architectures for Cognitive Development and Open-Ended Learning, Embodiment, Sensorimotor development
Abstract: Behavioral studies suggest that human cognitive development begins in the early developmental stage. However, owing to technical and ethical difficulties, there is limited knowledge on how embodied interaction contributes to early human development. Accordingly, constructive approaches for deepening the understanding of cognitive development are gaining attention in developmental robotics and dynamic systems. In this study, we performed a biologically plausible simulation of early human development, by updating the fetal simulation model that was developed in our previous study including a musculoskeletal body and uterine environment model. First, we updated the dynamics of the joints and muscles. Furthermore, we developed a new method to create soft objects of the desired shape in a rigid body simulator to achieve a soft uterine model. Subsequently, in a new simulation, we examined the impact of the soft uterine environment on the tactile experiences and subsequent cortical learning, compared to the extant uterine model and extrauterine environment. We observed that the soft uterine environment provided frequent and varied tactile information, which facilitated cortical learning toward achieving a higher cortical response to sensory inputs. Furthermore, we demonstrated that the embodied structure induced by the tactile sensor arrangement was crucial for cortical learning. Because actual fetuses and infants participate in flexible interactions, the proposed simulation using the soft environment could illuminate developmental care in the medical field.

Keywords: Sensorimotor development, Haptic and tactile perception, General Principles of Development and Learning
Abstract: The arm postures that infants adopt in the early months of life set up a repertoire of movement patterns that may aid in the development of reaching. There is evidence of tightly flexed arm postures in the womb, but how and when arm postures change over time after birth has not been systematically documented. The present study followed infants while lying in supine weekly from 3-weeks-old until they acquired head control. We documented the frequency rate of different steady state arm postures occurring when the hands were in contact with the body or the supporting surface. Across the observed developmental period, rates of flexed arm postures at the elbow decreased and more extended elbow arm postures increased but only around the time infants began to control their head. Initial elbow flexions, with the hands mostly oriented towards the head were superseded by elbow extensions with the hands primarily oriented towards the feet. Finally, most steady state arm postures entailed the forearm resting on the supporting surface or on the body rather than being held with the elbow in the air. Together, these findings show that arm postures adopted in the womb carry over into the early postnatal months and last for several weeks before extended arm postures become more prevalent. These findings have potential implications for the development of reaching and for preparing infants to produce arm movements away from the body.

Keywords: Sensorimotor development, Intrinsic Motivation, Exploration and Play, Body schema and body image
Abstract: Children are not born tabula rasa. However, interacting with the environment through their body movements in the first months after birth is critical to building the models or representations that are the foundation for everything that follows. We present longitudinal data on spontaneous behavior of three infants observed between about 8 and 25 weeks of age in supine position. We combined manual scoring of video recordings with an automatic extraction of motion data in order to study infants' behavioral patterns and developmental progression such as: (i) spatial distribution of self-touches on the body, (ii) spatial patterns and regularities of hand movements, (iii) midline crossing, (iv) preferential use of one arm, and (v) dynamic patterns of movements indicative of goal-directedness. From the patterns observed in this pilot data set, we can speculate on the development of first body and peripersonal space representations. Several methods of extracting 3D kinematics from videos have recently been made available by the computer vision community. We applied one of these methods on infant videos and provide guidelines on its possibilities and limitations---a methodological contribution to automating the analysis of infant videos. In the future, we plan to use the patterns we extracted from the recordings as inputs to embodied computational models of learning of body representations in infancy.

Keywords: Embodiment, Architectures for Cognitive Development and Open-Ended Learning, Baby robots
Abstract: Developmental Robotics models provide useful tools to study and understand the language learning process in infants and robots. These models allow us to describe key mechanisms of language development, such as statistical learning, the role of embodiment, and the impact of the attention payed to an object while learning its name. Robots can be particularly well suited for this type of problems, because they cover both a physical manipulation of the environment and mathematical modeling of the temporal changes of the learned concepts. In this work we present a computational representation of the impact of embodiment and attention on word learning, relying on sensory data collected with a real robotic agent in a real world scenario. Results show that the cognitive architecture designed for this scenario is able to capture the changes underlying the moving object in the field of view of the robot. The architecture successfully handles the temporal relationship in moving items and manages to show the effects of the embodied attention on word-object mapping.

Keywords: Language acquisition, Speech perception and production, Human-human and human-robot interaction and communication
Abstract: Infants participate in vocal coordination with others early in their life, even before they can rely on linguistic cues. They react sensitively to caregiver vocalizations, for instance, by imitating the caregiver and/or modulating their own vocalizations. When talking to an infant, caregivers also modulate their vocalizations, e.g., talk more slowly or with exaggerated prosody, which might attract infants’ attention and increase the clarity of vocal information. However, it is still unclear to what extent both parties’ vocal modifications dynamically influence each other. In this study, we investigate infants’ and caregivers’ vocal coordination in their daily interactions by applying multi-scale analysis on a global scale (i.e., a day), a middle scale (i.e., a conversational block), and a local scale (i.e., a turn). The day-long auditory recording data of nine infants, ages two months to three years, and their caregivers were analyzed. The results revealed that infants’ and caregivers’ vocalizations are differently coordinated on each timescale. On a global scale, infants and mothers react sensitively to each other’s vocalizations. Their conversation length varies across a day with a decreasing tendency. On a middle scale, infant-caregivers’ prosodic alignments increase over multiple turns in a conversation, indicating a continuous influence between them. Finally, more fine-grained analyses found that pitch-related features and pitch contours are aligned in each turn. The multi-scale analysis reveals the complexity of infant-caregiver interaction in the natural social environment, which inspires us to investigate the benefits of alignment in infants’ language learning at different timescales.

Keywords: Machine Learning methods for robot development, Human-human and human-robot interaction and communication, Language and semantic reasoning
Abstract: This paper focuses on robotic reinforcement learning with sparse rewards for natural language goal representations. An open problem is the sample-inefficiency that stems from the compositionality of natural language, and from the grounding of language in sensory data and actions. We address these issues with three contributions. We first present a mechanism for hindsight instruction replay utilizing expert feedback. Second, we propose a seq2seq model to generate linguistic hindsight instructions. Finally, we present a novel class of language-focused learning tasks. We show that hindsight instructions improve the learning performance, as expected. In addition, we also provide an unexpected result: We show that the learning performance of our agent can be improved by one third if, in a sense, the agent learns to talk to itself in a self-supervised manner. We achieve this by learning to generate linguistic instructions that would have been appropriate as a natural language goal for an originally unintended behavior. Our results indicate that the performance gain increases with the task-complexity.

Keywords: Human-human and human-robot interaction and communication
Abstract: Autonomous vehicle (AV) is progressing rapidly, but there are still many shortcomings when interacting with humans. To address this problem, it is necessary to study the human behaviors in human-AV interactions, and build a predictive model of human decision-making in the interaction. In turn, modelling human behavior in human-AV interaction can help us better understand human perception of AVs and human driving strategies. In this work, we first train multi-level AV agents using reinforcement learning (RL) models to imitate three mentalizing levels (i.e., level-0, level-1, and level-2), and then design a human-AV driving task that subjects interact with each level of AV agents in a two-lane merging scenario. Both human and AV driving behaviors are recorded. We found that conservative subjects obtain more rewards because of the randomness of the RL agents. Our results indicate that (i) human driving strategies are flexible and changeable, which allows to quickly adjust the strategy to maximize the reward when gaming against AV; (ii) human driving strategies are related to mentalizing ability, and subjects with higher mentalizing scores drive more conservatively. Our study shed light on the relationship between human driving policy and mentalizing in human-AV interactions, and it can inspire the next-generation AV.

Keywords: Reward and Value Systems, Prediction, planning and problem solving
Abstract: When designing and deploying Reinforcement Learning (RL) algorithms, one typically selects a single value for the discount rate, which results in an agent that will always be equally reactive or deliberative. However, similarly to humans, RL agents can benefit from adapting their planning horizon to the current context. To enable this, we propose the algorithm, RADAR: Reactive and Deliberate Adaptive Reasoning. Through experimental observations in episodic trajectories, RADAR enables an agent to choose a level of deliberation and reactivity adaptively according to the state it is in, given that there are cases where one mode of operation is better than the other. Through experiments in a grid world, we verify that the RADAR agent demonstrates a capability to adapt its reasoning modality to the current context. In addition, we found the RADAR agent exhibits different preferences regarding its thinking modes when a penalty for mental effort is included in its mathematical formulation.

Keywords: Human-human and human-robot interaction and communication, Social robots and social learning, Models of emotions and internal states
Abstract: When interacting with others in our everyday life, we prefer the company of those who share with us the same desire of closeness and intimacy (or lack thereof), since this determines if our interaction will be more o less pleasant. This sort of compatibility can be inferred by our innate attachment style. The attachment style represents our characteristic way of thinking, feeling and behaving in close relationship, and other than behaviourally, it can also affect us biologically via our hormonal dynamics. When we are looking how to enrich human-robot interaction (HRI), one potential solution could be enabling robots to understand their partners' attachment style, which could then improve the perception of their partners and help them behave in an adaptive manner during the interaction. We propose to use the relationship between the attachment style and the cortisol hormone, to endow the humanoid robot iCub with an internal cortisol inspired framework that allows it to infer participant's attachment style by the effect of the interaction on its cortisol levels (referred to as R-cortisol). In this work, we present our cognitive framework and its validation during the replication of a well-known paradigm on hormonal modulation in human-human interaction (HHI) - the Still Face paradigm.

Keywords: Human-human and human-robot interaction and communication
Abstract: This pilot study is one of the first to investigate child-robot interaction (CRI) using the intergroup contact procedure. We examined how a brief positive intergroup contact with a humanoid robot (NAO) affects children’s attitudes towards robots. To evaluate young children’s attitudes towards humanoid robots, we tested 39 children (4-6 years old) in an experimental design comparing the interaction condition and the no-interaction condition. Results indicate that, unlike adults in previous studies, our child participants consistently exhibited positive attitudes towards the robot regardless of the condition, but children in the interaction condition favored the robot over an ingroup peer more strongly than did children in the no-interaction condition. We discuss the possibility that young children see robots as an admired outgroup and favor them over their ingroup members. Our findings provide important insights into the status of humanoid robots as evaluated by young children.

Keywords: Human-human and human-robot interaction and communication, Social robots and social learning, Models of human motion and state
Abstract: In this study, we investigate the possible effects on the human movement kinematics of the presence of a humanoid robot during an interaction. We conducted an experiment in which 11 participants were required to grab a cube and to drop it in the hands of a robot, a human and in a rectangular box. Using this setup, we explore whether the kinematics of the participants' gestures would be particularly influenced by the presence of the robot and whether this influence would be due to the fact that the robot is considered as a possible social partner. The results show that the condition that includes the robot partner leads to kinematic modulation that are similar to the condition including the human partner. Furthermore, there are significant differences between the condition including the robot and the one with the box. Finally, our results show that the participants pro-social behavior is correlated with the perceived agency of the robot as evaluated by a user questionnaire.

Keywords: Intrinsic Motivation, Exploration and Play
Abstract: Intrinsic motivation is a common method to facilitate exploration in reinforcement learning agents. Curiosity is thereby supposed to aid the learning of a primary goal. However, indulging in curiosity may also stand in conflict with more urgent or essential objectives such as self-sustenance. This paper addresses the problem of balancing curiosity, and correctly prioritising other needs in a reinforcement learning context. We demonstrate the use of the multi-objective reinforcement learning framework C-MORE to integrate curiosity, and compare results to a standard linear reinforcement learning integration. Results clearly demonstrate that curiosity can be modelled with the priority-objective reinforcement learning paradigm. In particular, C-MORE is found to explore robustly while maintaining self-sustenance objectives, whereas the linear approach is found to over-explore and take unnecessary risks. The findings demonstrate a significant weakness of the common linear integration method for intrinsic motivation, and the need to acknowledge the potential conflicts between curiosity and other objectives in a multi-objective framework.

Keywords: Intrinsic Motivation, Exploration and Play, Models of self and agency, Active learning
Abstract: Experience replay an essential role as an information-generating mechanism plays in reinforcement learning systems that use neural networks as function approximators. It enables the artificial learning agents to store their past experiences in a sliding-window buffer, effectively recycling them in the process of a continual re-training of a neural network. The intermediary process of experience caching opens a possibility for an agent to optimize the order in which the experiences are sampled from the buffer. This may improve the default standard, i.e., the stochastic prioritization based on Temporal-Difference error (or TD-error), which focuses on experiences that carry more Temporal-Difference surprise for the approximator. A notion of informed prioritization is proposed, a method relying on fast on-line confidence estimates of approximator predictions in order to be able to dynamically exploit the benefits of TD-error prioritization only when its prediction confidence about the selected experiences increases. The presented informed-stochastic prioritization method of replay buffer sampling, implemented as a part of standard staple Deep Q-learning algorithm outperformed the vanilla stochastic prioritization based on TD-error in 41 out of 54 trialed Atari games.

Keywords: Machine Learning methods for robot development, Action selection and planning, Statistical Learning
Abstract: This paper proposes a novel benchmark for 3D shape completion methods based on their adaptability for the task of robotic grasping.

Firstly, state-of-the-art single image shape completion methods are used to reconstruct object shapes from RGB images. These images contain views of objects belonging to different categories. Two specific shape-reconstruction methods are selected for this study.

On the next step, the resulting 3D reconstructions of these methods are loaded into a robotic grasp simulator in order to attempt to grasp the objects from different approaches and using different hand configurations. Then, the unsuccessful grasps~(according to a grasp quality metric) are excluded and the remaining ones are used to compute a grasp related metric, the Joint Error, which evaluates the usability of the reconstructed mesh for grasping the ground-truth 3D model.

Finally, based on the results obtained from our experiments, we draw several conclusions about the performance of each of the methods. Furthermore, an analysis is made for the possible correlation between the newly proposed Joint Error metric and the popular reconstruction quality metrics used by most shape completion methods. Our results indicate that geometry-based reconstruction metrics are mostly inadequate for assessing the usability of a 3D reconstruction algorithm for robotic grasping.

Keywords: Machine Learning methods for robot development, Cogntive vision, Human-human and human-robot interaction and communication
Abstract: Nowadays, engagement detection plays an essential role in e-learning education and robotics. In the field of human-agent interaction, it is of great interest to know the attitude of the human peer towards the interaction so that the agent can react accordingly. The goal of this paper is to develop an automatic real-time engagement recognition system using a combination of non-verbal features (gaze direction, head position, facial expression and distance between users) extracted using computer vision techniques. Our system uses a machine learning model based on Random Forest and achieves 86 accuracy. Furthermore, using an RGB camera, the system can detect the level of user engagement in real-time and classify it into four levels of intensity.

Keywords: Machine Learning methods for robot development, General Principles of Development and Learning
Abstract: Variational Autoencoders (VAEs) have attracted a lot of attention from the machine learning community in recent years. The usage of VAEs in learning by demonstration and robotics is still very restricted due to the need for effective learning from only a few examples and due to the difficult evaluation of the reconstruction quality. In this paper, we utilize the current models of conditional variational autoencoders for the purpose of teaching a robot simple actions from demonstration in an incremental fashion. We in detail evaluate various training approaches and define parameters that are important for enabling high-quality samples and reconstructions. The quality of the generated samples in different stages of learning is evaluated both quantitatively and qualitatively on the humanoid robot Pepper. We show that the robot can reach a reasonable quality of generated actions already after 20 observed samples.

Keywords: Machine Learning methods for robot development, Prediction, planning and problem solving, Robot prototyping of human and animal skills
Abstract: This work investigates the complications of robotic learning using reinforcement learning (RL). While RL has enormous potential for solving complex tasks its major caveat is the computation cost- and time-intensive training procedure. This work aims to address this issue by introducing a human-like thinking and acting paradigm to a RL approach. It utilizes model-based deep RL for planning (think) coupled with inverse kinematics (IK) for the execution of actions (act). The approach was developed and tested using a Franka Emika Panda robot model in a simulated environment using the PyBullet physics engine Bullet. It was tested on three different simulated tasks and then compared to the conventional method using RL-only to learn the same tasks. The results show that the RL algorithm with IK converges significantly faster and with higher quality than the applied conventional approach, achieving 98%, 99% and 98% success rates for tasks 1-3 respectively. This work verifies its benefit for use of RL-IK with multi-joint robots.