Keywords: Motion and Path Planning, Decision Making Systems, Systems for Field Applications
Abstract: To work in unknown outdoor environments, autonomous sampling machines need the ability to target samples despite limited visibility and robotic arm reach distance. We design a heuristic guided search method to speed up the search process and more efficiently localize the approximate center of soil regions. Through simulation experiments, we assess the effectiveness of the proposed algorithm and discover superior performance in terms of speed, distance traveled, and success rate compared to naive baselines.

Keywords: Integration Platform, Factory Automation, Vision Systems
Abstract: Localization is a fundamental requirement for a wide range of robotic applications, but existing systems often require complex, resource-intensive and costly setups. We propose a cost-effective localization system that integrates multiple fiducial markers and multiple cameras for enhancing both pose estimation accuracy, detection range and frequency while reducing costs and providing camera placement flexibility. Our system reduces position RMSE from 13.45 to 3.6 centimeters (73% improvement) and can achieve 100% detection coverage while leveraging 3 to 5 cameras instead of 10, no IMU or odometry compared to our previous single-marker multi-camera system, MoCArU. When tested at different camera heights, our system outperforms the previous one in all evaluated conditions. It also increases the frequency of estimates, as determined by a qualitative analysis. Additionally, we evaluate various pose fusion methods, demonstrating that a simple and quick mean-based approach effectively maintains tracking accuracy with our system. This flexible, low-cost system provides a reliable and practical solution for indoor localization, making it a valuable option for various indoor tracking and monitoring applications.

Keywords: Autonomous Vehicle Navigation, Motion and Path Planning, Path Planning for Multiple Mobile Robots or Agents
Abstract: This paper introduces a motion control algorithm for an autonomous four-wheel steering(4WS) vehicle system, which independently controls the steering angles of the front and rear wheels. This capability allows for three distinct driving modes: front steering mode, crab steering mode, and symmetrical steering mode. The proposed algorithm effectively uses these modes to achieve precise maneuverability, facilitating accurate navigation toward the target location. We conducted a kinematic analysis of the four-wheel steering vehicle, with a focus on its motion characteristics. A simple motion control law is developed using the kinematic analysis and the steering mode transition of the 4WS vehicle. To validate the feasibility and effectiveness of the proposed algorithm, initial numerical simulation results are presented.

Keywords: Autonomous Vehicle Navigation, Intelligent Transportation Systems, Robotic hands and grasping
Abstract: This paper presents a feasibility study, including simulations and prototype tests, on the autonomous operation of a multi-limbed intra-vehicular robot (mobile manipulator) designed to assist astronauts with logistical tasks on the International Space Station (ISS). Astronauts spend significant time on tasks such as preparation, close-out, and the collection and transportation of goods, reducing the time available for critical mission activities. Our study explores the potential for a mobile manipulator to support these operations, emphasizing the need for autonomous functionality to minimize crew and ground operator effort while enabling real-time task execution. We focused on the robot's transportation capabilities, simulating its motion planning in 3D space. The actual motion execution was tested with a prototype on a 2D floating table to mimic a microgravity environment. The results demonstrate the feasibility of performing these tasks with minimal human intervention, offering a promising solution to enhance operational efficiency on the ISS.

Keywords: Systems for Field Applications, Systems for Search and Rescue Applications, Network Systems
Abstract: When several robots operate in the same workspace simultaneously, e.g., in nuclear decommissioning, it is necessary to consider their movement in narrow areas. In such environments, both autonomous navigation and manual remote operation are necessary because of the unknown operating conditions that may require human judgements. We develop and experimentally verify an enhanced remotely operated robot system that can freely switch between autonomous navigation and manual remote operation with a common communication protocol; its operating conditions can be changed according to the workspace characteristics. In the proposed system, a robot could move in narrow areas by specifying the workspace and automatically changing its obstacle detection distance. Furthermore, it showed that two robots could pass-by each other without any coordination by appropriate conditions. We will investigate additional parameters, such as movement speed and priority using inter-robot communication for nuclear decommissioning and other applications.

Keywords: Autonomous Vehicle Navigation, Automation Systems, Systems for Field Applications
Abstract: The increasing use of drones for maritime inspections in near-coastal areas has highlighted the need for more sophisticated tools that can assist operators in optimizing mission planning and flight paths. Despite advancements in drone technology, challenges remain in efficiently combining available data to enhance range, increase available inspection time, and overall mission effectiveness. This paper proposes a novel application that integrates Automatic Identification System (AIS) data into the drone navigation and mission planning process. By leveraging real-time AIS data, including ship positions, speeds, and headings, the system provides dynamic waypoint suggestions to operators, facilitating more precise and efficient path planning. Additionally, the integration of wind correction factors into the flight planning algorithm further enhances battery management, allowing for extended mission durations. Through simulations and real-world experiments, the proposed system demonstrates significant improvements in mission planning efficiency, drone range, and inspection time, offering a promising solution for the growing demands of maritime inspections

Keywords: Motion and Path Planning, Control Theory and Technology, Autonomous Vehicle Navigation
Abstract: Model-based control is a crucial component of robotic navigation. However, it often struggles with entrapment in local minima due to its inherent nature as a finite, myopic optimization procedure. Previous studies have addressed this issue but sacrificed either solution quality due to their reactive nature or computational efficiency in generating explicit paths for proactive guidance. To this end, we propose a motion planning method that proactively avoid local minima without any guidance from global paths. The key idea is repulsive potential augmentation, integrating high-level directional information into the Model Predictive Path Integral control as a single repulsive term through an artificial potential field. We evaluate our method through theoretical analysis and simulations in environments with obstacles that induce local minima. Results show that our method guarantees the avoidance of local minima and outperforms existing methods in terms of global optimality without decreasing computational efficiency.

Keywords: Software, Middleware and Programming Environments, Factory Automation, Automation Systems
Abstract: In this paper we introduce an open source cross-platform C++ interface for controlling Universal Robot (UR) manipulators. This interface is capable of controlling all UR robots that facilitates a RTDE (Real-time Data Exchange), which is the communication protocol used by this interface. Previous interfaces did not leverage the RTDE and it was one of the motivating factors for writing this interface. ur_rtde can be used both from C++ and Python with bindings. A Python package has been released, to make it easy to install and use on Windows, Linux and MacOS. We show that the proposed interface outperforms other interfaces in terms of real-time characteristics.

Keywords: Software, Middleware and Programming Environments, Software Platform, Integration Platform
Abstract: This paper presents a multitasking system for low-level microcontrollers. The main focus in developing this operating system was to design an optimized software that does not have high requirements in terms of memory or computing time. Another requirement was to be able to port the system to different platforms very easily. Therefore, C was chosen as the programming language. Another focus was on the ease of use of the program. It should also be able to be used by less experienced programmers without a long training phase. The program is now used in a wide variety of projects, from robot systems to wide range of other applications, on different microcontrollers, such as Xmega256A3 or AVR128DB48.

Keywords: Software Platform, Integration Platform, Multi-Robot Systems
Abstract: Recent advancements in computing, network speeds, and cloud technology have led to the emergence of Software-Defined x (SDx), a concept that promotes a shift from traditional hardware-dependent systems to more flexible, software-defined environments. In the context of robotics, this approach is termed Software-Defined Robotics (SDR), which aims to abstract hardware functions into software, enabling dynamic development and deployment of robotic applications without the need for extensive hardware modifications. This paper proposes the architecture of SDR to enhance flexibility, interoperability, and portability in robotic systems, which is designed based on the information models defined in ISO/DIS 22166-202 and ISO 22166-201. This paper demonstrates the feasibility of SDR using the TurtleBot3 as a case study. The proposed approach is validated by showing two similar but distinct SDR implementations for the TurtleBot3 through Gazebo simulation. The results emphasize the potential of SDR to revolutionize the development and management of robotic systems by reducing hardware dependency and fostering innovation. This paper provides a comprehensive overview of the architecture, advantages, and constraints of SDR and presents an implementation model based on the TurtleBot3, providing insights into future developments in software-defined robotics.

Keywords: Network Systems, Autonomous Vehicle Navigation, Integration Platform
Abstract: Trust is increasingly becoming a key consideration in the design of autonomous robotic systems. In industrial applications, security and trust in the system are requirements for widespread adoption. Blockchain technologies have emerged as a potential solution to address identity management and secure data aggregation and control. However, the vast majority of works to date utilize Ethereum and smart contracts that are not scalable or well suited for industrial applications. This paper presents what is, to the best of our knowledge, the first integration of ROS 2 with the Hyperledger Fabric blockchain. With a framework that leverages Fabric smart contracts and ROS 2 through a Go application, we delve into the potential of using blockchain for controlling robots, and gathering and processing their data. We demonstrate the applicability of the proposed framework to an inventory management use-case where different robots are used to detect objects of interest in a given area. Designed to meet the requirements of distributed robotic systems, we show that the performance of the robots is not impacted significantly by the blockchain layer. At the same time, we provide examples for developing other applications that integrate Fabric smart contracts with ROS 2. Our results pave the way for further adoption of blockchain technologies in autonomous robotic systems for building trustable data sharing.

Keywords: Software Platform, System Simulation, Decision Making Systems
Abstract: This paper presents an open-source, lightweight, yet comprehensive software framework, named RPC, which integrates physics-based simulators, planning and control libraries, debugging tools, and a user-friendly operator interface. RPC enables users to thoroughly evaluate and develop control algorithms for robotic systems. While existing software frameworks provide some of these capabilities, integrating them into a cohesive system can be challenging and cumbersome. To overcome this challenge, we have modularized each component in RPC to ensure easy and seamless integration or replacement with new modules. Additionally, our framework currently supports a variety of model-based planning and control algorithms for robotic manipulators and legged robots, alongside essential debugging tools, making it easier for users to design and execute complex robotics tasks. The code and usage instructions of RPC are available at https://github.com/shbang91/rpc.

Keywords: Integration Platform, System Simulation, Software Platform
Abstract: Greenhouses allow for local food production, protecting plants from the external environment and maintaining optimal growing conditions. This paper presents a modelling methodology for integrating renewable energy and storage into community-scaled greenhouse operations. By adopting a heating demand model for building envelopes, the capabilities of energy modelling tool OSeMOSYS can be extended to represent the energy system of such a greenhouse. Using real solar and temperature hourly data, the results capture meaningful aspects of the design, operation, and costs that may otherwise be lost in averaged modelling approaches.

Keywords: Automation Systems, Vision Systems
Abstract: The rise of self-driving laboratories has seen significant growth across various research domains, particularly in chemistry, materials science and life science. However, a major challenge persists---the majority of self-driving systems are costly due to the use of highly precise lab equipment, robotic platforms, and case-specific algorithms, rendering these systems less accessible for educational purposes. This paper takes a multidisciplinary approach; we first introduce a small-scale self-driving experiment platform tailored for educational use, focusing on liquid materials mixing tasks commonly seen in chemistry and life sciences. To understand the operational status in real-time while maintaining self-driving capability and efficiency, we propose a novel system concept: employing a mobile robot as the lab supervisor to monitor the experiment process across multiple identical self-driving platforms. Specifically, this paper focuses on implementing a vision-based monitoring system. A deep learning architecture with a new training strategy is presented to jointly address two tasks: (a) vessel and content material segmentation and (b) volume estimation. The two tasks can be trained independently but can be inferred end-to-end by integrating them into the Mask R-CNN framework. Through evaluating the monitoring module on a real dataset, the results showcase promising detection capabilities, good real-time performance, and compatibility with the self-driving platform, indicating the feasibility of our proposed system.

Keywords: Network Systems, Control Theory and Technology, Human-Robot/System Interaction
Abstract: Teleoperation of remote assist robots has recently advanced owing to the development of communication technology. However, anonymous malicious attacks may intercept or falsify the network control system. Therefore, it is necessary to improve the security against cyber-attacks. As a countermeasure, an encrypted control method has been applied to prevent intercepting and detect the falsification of control parameters, as well as control signals by performing control operations with encrypted signals and parameters. Furthermore, we have proposed the algorithm to detect and restore the attack exploiting a vulnerability of encryption, which falsifies the plaintext by multiplying a constant factor. However, the algorithm is only effective for periodic motions with a fixed operating frequency and amplitude. If the frequency or amplitude of the motion is smaller than those of the base motion, the algorithm cannot detect the attack. To solve the problem, we propose an improved algorithm to detect and restore the attack in periodic motion with variable operating frequency and amplitude. In the proposed method, the operating frequency and amplitude are obtained through the position frequency analysis. They contribute to update the base energy, which is the average energy of corresponding periodic motion. We verified the proposed method for the bilateral control system using ElGamal encryption and experimentally confirmed its effectiveness against the FDI attack in various periodic motion.

Keywords: Motion and Path Planning, Systems for Search and Rescue Applications, Multi-Modal Perception
Abstract: This paper details a system for fast visual exploration and search without prior map information. We leverage frontier based planning with both LiDAR and visual sensing and augment it with a perception module that contextually labels points in the surroundings from wide Field of View 2D LiDAR scans. The goal of the perception module is to recognize surrounding points more likely to be the search target in order to provide an informed prior on which to plan next best viewpoints. The robust map-free scan classifier used to label pixels in the robot's surroundings is trained from expert data collected using a simple cart platform equipped with a map-based classifier. We propose a novel utility function that accounts for the contextual data found from the classifier. The resulting viewpoints encourage the robot to explore points unlikely to be permanent in the environment, leading the robot to locate objects of interest faster than several existing baseline algorithms. Our proposed system is further validated in real-world search experiments for single and multiple search objects with a Spot robot in two unseen environments. Videos of experiments, implementation details and open source code can be found at https://sites.google.com/view/lives-2024/home.

Keywords: Automation Systems, Vision Systems, Machine Learning
Abstract: In the field of robotics, the construction of advanced perception models is essential for the successful execution of complex tasks. Traditional perception models, often grounded in cognitive frameworks, fall short in adequately processing and interpreting visual data. There is a pressing need to enhance these models with advanced visual processing capabilities. The integration of sophisticated vision models with cognitive frameworks is expected to significantly enhance the performance of perception models, yet such integrations remain underexplored. In this paper, we propose a novel Vision-Aware Cognition Model that effectively merges visual and cognitive components to advance robot perception. Our model integrates a cognition model, which incorporates contextual memory for nuanced long-term memory and context comprehension, with a vision model that employs spatial attention to focus on key regions of visual input. This harmonious integration enables not only robust feature extraction but also heightened adaptability to visual environmental changes. We evaluated our model using a simulated robotic hand on a valve-turning manipulation task. By leveraging saliency visualization, we made the robot's decision-making process transparent, showcasing the distinct functions of the visual and cognitive components. The vision model demonstrates superior object segmentation, while the cognition model is adept at operation points tracking. By leveraging the strengths of both components, the proposed model achieves efficient hybrid feature extraction. Furthermore, we conducted quantitative evaluations of the model's adaptability to various visual changes, which revealed statistically significant performance improvements, highlighting its remarkable capacity for enhancing robot perception.

Keywords: Vision Systems, Intelligent and Flexible Manufacturing, Factory Automation
Abstract: The integration of robotics in the garment industry remains limited, primarily due to the challenges in the highly deformable nature of garments. This study thus explores a vision-based garment and garment parts recognition model to facilitate the application of robots in garment manipulation. The main objective is to detect and segment each garment piece from a random table and provide multi-dimensional information on it, as well as recognize garment parts such as collar, sleeves, and main body part, to facilitate generating grasping points for various robotic tasks. An MTL (Multi-Task Learning) model based on YOLOv8 and HyCTAS’s self-attention head is proposed for this. Transfer learning is applied and the model is fine-tuned and tested on a self-collected dataset as well as an open-source garment dataset Fashionpedia. Experiment results demonstrate that this MTL model is able to substantially improve the processing speed while having a minimal decrease in mask average precision for each integrated vision task. And while this performance preservation is mainly attributed to the HyCTAS implementation, further enhancements can be achieved by adding auxiliary tasks and loading weights from single tasks.

Keywords: Automation Systems, Vision Systems, System Simulation
Abstract: Laboratory processes involving small volumes of solutions and active ingredients are often performed manually due to challenges in automation, such as high initial costs, semi-structured environments and protocol variability. In this work, we develop a flexible and cost-effective approach to address this gap by introducing a vision-based system for liquid volume estimation and a simulation-driven pouring method particularly designed for containers with small openings. We evaluate both components individually, followed by an applied real-world integration of cell culture automation using a UR5 robotic arm. Our work is fully reproducible: we share our code at at https://github.com/DaniSchober/LabLiquidVision and the newly introduced dataset LabLiquidVolume is available at https://data.dtu.dk/articles/dataset/LabLiquidVision/251031 02.

Keywords: Vision Systems, Automation Systems, Machine Learning
Abstract: In this paper, we study the problem of direct trajectories in visual control of nonholonomic mobile robots. To address this challenge, we propose combining deep reinforcement learning with panoramic vision to learn a control policy that maps input images to control velocities. We demonstrate that the adopted approach can precisely drive the robot to a desired pose. Additionally, it enables the emergence of various control strategies to achieve interesting trajectories. Our approach is validated and evaluated in simulation and transferred to the real world.

Keywords: Vision Systems
Abstract: Recently, Object-oriented SLAM(Object SLAM) has attracted extensive research due to its ability to perceive the environment at a 3d object level. Existing object SLAM methods mostly focus on constructing 3d object map for static objects or mitigating the impact of currently dynamic objects on localization and mapping. However, detection of semi-static objects whose position change while unobserved still pose a significant challenge, resulting in outdated maps, which could lead to localization and robot application failures. In this paper, we propose a method to compare current observation with the existing map, enabling the continuous detection and updating of outdated sections within the map caused by position-changing semi-static objects. First, we introduce Object Covisibility Graph(OCG) to maintain the historically observed co-visibility relationships between objects. Building on this, we design an algorithm that uses the OCG to determine whether the current camera is within the observable region of each object, and subsequently implement an object state updating algorithm to detect and update outdated sections continuously. We conduct experiments on our self-make dataset with changing objects and a dataset with only static objects. The experimental results show that our method updates the outdated parts of the map more effectively compared to previous studies.

Keywords: Software, Middleware and Programming Environments, Multi-Modal Perception, Systems for Service/Assistive Applications
Abstract: This paper introduces a novel pipeline to enhance the precision of object masking for robotic manipulation within the specific domain of masking products in convenience stores. The approach integrates two advanced AI models, CLIP and SAM, focusing on their synergistic combination and the effective use of multimodal data (image and text). Emphasis is placed on utilizing gradient-based attention mechanisms and customized datasets to fine-tune performance. While CLIP, SAM, and Grad-CAM are established components, their integration within this structured pipeline represents a significant contribution to the field. The resulting segmented masks, generated through this combined approach, can be effectively utilized as inputs for robotic systems, enabling more precise and adaptive object manipulation in the context of convenience store products.

Keywords: Automation Systems, Intelligent and Flexible Manufacturing, Vision Systems
Abstract: For the purpose of performing robotic assembly from a novel graphical instruction, this paper proposes a new method for aligning assembly parts based on the visual information guided by the image in a graphical instruction manual. Our proposed method comprises two phases: We first detect an assembly part drawn in the instruction manual and then estimate its relative pose among multiple assembly parts to be assembled. For the detection of assembly parts, we build the matching algorithm based on fast-directional chamfer matching (FDCM) by utilizing multiple images of actual assembly parts captured from multiple angles. For the relative pose estimation, we collect the poses of the identified parts and match them with the assembly scenes in the graphical instruction manuals. We conducted collaborative assembly experiments between a human and a robot for a set of furniture parts. We confirmed that the captured images matched the graphics contained in the assembly manual well. In addition, we confirm that, with the help of a human, a robot can efficiently assemble furniture with a novel instruction manual.

Keywords: Decision Making Systems, Motion and Path Planning, Automation Systems
Abstract: To fully integrate robots into household settings, they must be capable of autonomously planning and executing diverse tasks. However, task and motion planning for multi-step manipulation tasks remains an open challenge in robotics, especially for long-horizon tasks in dynamic environments. This study presents an integrated task and motion planning (TAMP) robotic framework for real-world cooking tasks using a dual-arm robotic system. Our framework combines PDDLStream, an existing TAMP framework, with the MoveIt Task Constructor, a multi-stage manipulation planner, to improve multi-step motion planning for long-horizon tasks. We enhance our framework with various cooking-related skills, including object fixturing, force-based tip detection, and slicing using Reinforcement Learning (RL). As a motivating case study, we address the long-horizon task of preparing a simple cucumber salad, involving slicing and serving it on a plate. We showcase our framework through both simulation and real robot demonstration.

Keywords: Mechanism Design, Mechatronics Systems, Hardware Platform
Abstract: In robotics research that uses actuators to replace actual work in system integration, there is a growing demand for servo modules that we can use in various robots. We require the module to have the following three functions. 1) Supporting its weight without using energy, 2) the function to move the output link by external manipulation, and 3) being inexpensive and easy to duplicate and combine. In this study, we developed servo modules with openable worm gear reduction mechanisms ready for mass production to support a large load in the driven state and to move passively in the free state. As example configurations, we show 1: a winch for suspending a robot, 2: a non-backdrivable hand, 3: a teaching device, and 4: a carrying cart that can support its weight and carry heavy objects, demonstrating that the module can be easily used in a variety of configurations while fulfilling its intended functions.

Keywords: Plant Engineering, Human-Robot/System Interaction, Multi-Robot Systems
Abstract: This paper presents a novel concept for a reconfigurable robotic system specifically designed to meet the demands of hybrid integration for miniaturized photonic and quantum System-in-Packages (SiPs). The proposed solution introduces a distinctive approach to ultra-high-resolution, multi-telerobotic assembly and inspection. By integrating key Industry 5.0 principles, it establishes a human-centric control framework that minimizes both physical and cognitive stress while ensuring the human operator remains in full control at all times. The robotic system features eight robots working simultaneously within a compact footprint of just 5 x 10 cm^2. A comprehensive digital twin framework constitutes a central element of the robotic system. It encompasses the robotic workcell, the SiP under assembly, and the components to be integrated, ensuring precise adherence to design specifications. Key functionalities include automated path planning in a multi-robotic environment, collision avoidance in a densely packed workcell, and virtual fixtures to guide teleoperation, enhancing the operator's control and interaction through advanced and intuitive human-machine interfaces (HMI). The proposed system meets the critical demands of ultra-high-resolution assembly for complex, high-value SiPs, providing high flexibility and ease of operation for small-batch manufacturing.

Keywords: Automation Systems, Autonomous Vehicle Navigation, Intelligent Transportation Systems
Abstract: This paper presents an efficient solution to 3D-LiDAR-based Monte Carlo localization (MCL). MCL robustly works if particles are exactly sampled around the ground truth. An inertial navigation system (INS) can be used for accurate sampling, but many particles are still needed to be used for solving the 3D localization problem even if INS is available. In particular, huge number of particles are necessary if INS is not available and it makes infeasible to perform 3D MCL in terms of the computational cost. Scan matching (SM), that is optimization-based localization, efficiently works even though INS is not available because SM can ignore movement constraints of a robot and/or device in its optimization process. However, SM sometimes determines an infeasible estimate against movement. We consider that MCL and SM have complemental advantages and disadvantages and propose a fusion method of MCL and SM. Because SM is considered as optimization of a measurement model in terms of the probabilistic modeling, we perform measurement model optimization as SM. The optimization result is then used to approximate the measurement model distribution and the approximated distribution is used to sample particles. The sampled particles are fused with MCL via importance sampling. As a result, the advantages of MCL and SM can be simultaneously utilized while mitigating their disadvantages. Experiments are conducted on the KITTI dataset and other two open datasets. Results show that the presented method can be run on a single CPU thread and accurately perform localization even if INS is not available.

Keywords: Human-Robot/System Interaction, Machine Learning, Integration Platform
Abstract: The increasing demand for mobile robots in indoor environments such as hospitals, offices, and residential buildings has highlighted the need for affordable, privacy-preserving navigation and interaction capabilities. This study introduces a cost-effective, on-device BERT-based natural language navigation system that enables robots to interpret human commands into goals. The system is designed for deployment on lightweight embedded computers and updates without requiring model retraining, ensuring scalability and flexibility. We also propose an AprilTAg-augmented SLAM system to reduce navigation errors in common indoor challenges like ramps and transparent obstacles. Experiments in real-world settings statistically demonstrate that our solution significantly reduces errors in these scenarios, offering a more reliable approach to indoor robot navigation.

Keywords: Systems for Service/Assistive Applications, Integration Platform, Human-Robot/System Interaction
Abstract: Reduced vision significantly impacts the daily lives of people with visual impairments (PVI), often posing challenges in navigation and spatial awareness. To enhance the semantic reasoning capabilities of assistive technologies, we have developed a guidance system that integrates large vision-language models (VLMs) with a collision-avoidance drone. This system provides navigational assistance in indoor environments by interpreting semantic wayfinding signs. At the software level, we propose a hierarchical cross-prompt VLM (HCP-VLM) structure that leverages both Claude 3.5 Sonnet and ChatGPT 4o. This structure improves the reasoning accuracy of semantic wayfinding signs to 76.73%, outperforming the standalone accuracies of Claude (74.73%) and ChatGPT (66.35%). A specialized wayfinding sign dataset was developed to fine-tune and evaluate the VLM. At the hardware level, an ultralight dual-modal Time of Flight (TOF) Laser-Camera module was integrated into the drone to detect obstacles, track users, and identify signs. Additionally, a vibration module was designed to communicate orientation and mobility information to users. The system's performance was evaluated in unfamiliar office buildings with two blindfolded sighted subjects, both of whom successfully located their target rooms with assistance from the system. To further drive innovation, we have released the dataset and code for public access, aiming to inspire advancements in intelligent assistive technologies.

Keywords: Vision Systems, Control Theory and Technology, Sensor Fusion
Abstract: The vehicle navigating through narrow and crowded environments requires detailed shape information of the surrounding pedestrians for collision avoidance. While Light Detection and Ranging (LiDAR) is highly effective at at measuring the position of objects, its performance diminishes as the distance between the LiDAR and the objects increases. The number of data points acquired decreases, leading to a less informative reconstruction of the object's pose and shape. To address this issue, this study proposes a pedestrian tracking method that involves constructing multiple pedestrian models and estimating the appropriate model parameters from likelihoods using the Interacting Multiple Model, based on the number of point clouds. We prepare three models: ellipse, bounding box, and point cloud center of gravity models. The ellipse and bounding box models estimate pose and size, while the point cloud center of gravity model estimates pose. The elliptical model uses Random Sample Consensus to determine model parameters that suppress arm swing and body sway during walking. Through experimental validation, this method effectively demonstrated its ability to continuously track pedestrians, including those with only a few acquired data points from a pedestrian located far from the LiDAR, while accurately estimating their pose and size.

Keywords: Decision Making Systems, Vision Systems, Multi-Modal Perception
Abstract: This work presents a novel approach to zero-shot visual object goal navigation that leverages the ability of visual Large Language Model (vLLM) for finding target in unknown environment. Our system combines semantic mapping with vLLM-driven decision-making to direct robots towards target objects. The core of our approach lies in using vLLM to generate a value map between explored areas and the target object using cosine similarity based on prompt identically, incorporating both visual and semantic information from RGB-D image observations. This value map, along with a constructed semantic map and extracted movable frontier points, serves as a historic information for the vLLM to select one of the frontiers to explore next. We evaluate our method on two single-floor scenes from the Habitat-Matterport 3D dataset and Habitat Synthetic Scenes Dataset using the Habitat simulator separately. Our experiments demonstrate that the proposed approach has the potential to explore efficiently, particularly excelling when utilizing semantic information from simulator. The results show promise of our method in zero-shot navigation scenarios if overcome the common semantic extraction challenge. This work contributes to the growing field of language-driven exploration and exhibits how advanced large language model can effectively tackle complex navigation tasks.

Keywords: Autonomous Vehicle Navigation, Integration Platform, Sensor Fusion
Abstract: Accurate and robust pose estimation is essential for any autonomous vehicle. While sensors like GNSS, LiDAR, and camera are widely used for state estimation, they admit weaknesses under challenging environments e.g. poor satellite signals, low light, or adverse weather. Recent 3D and 4D radars have emerged as a robust alternative, providing sparse point clouds with information on the detected targets' range, angle, and radial velocity. As a result, interest in radar-based odometry (RO) has grown steadily. In this paper, we explore two aspects: radar-only ego-motion estimation and multi-radar-vehicle extrinsic calibration. We present a Doppler-based radar odometry algorithm using an Implicit Extended Kalman Filter and propose a novel, radar-only and target-less multi-radar-vehicle calibration method. An observability analysis is conducted to optimize sensor placement considering both the odometry and calibration. Finally, the proposed methods are validated through both simulation and real data.

Keywords: Autonomous Vehicle Navigation, Vision Systems, Logistics Systems
Abstract: To operate a large number of autonomous robots, an accurate and low-cost localization technology is recommended. To meet this demand, we have proposed an original localization approach, Visual-Geometric-Matching (VGM), using a monocular RGB camera and a pre-build map, and have developed it as the client-server localization system. In this paper, we propose a panorama-VGM method, which achieves efficient memory usage and efficient image matching. By applying a cylindrical panorama transform to the template images, the redundant information is eliminated, and the VRAM memory usage is significantly reduced. The panorama transform is also applied to query image from a client, and both images are matched on panoramic projection surface. Furthermore, we integrated the results of panorama-VGM with the odometry data by Extended Kalman Filter and utilize for a mobile robot navigation. Experimental results show that the proposed panorama-VGM is accurate, robust and practical localization system. Specifically: (1) the median error is 0.11 [m] in the single algorithm accuracy evaluation, (2) the median error is 0.12 [m] in the mobile robot navigation experiment compared with LiDAR + EMCL, (3) the travel distance is 2,590 [m] during a continuous 77-minute durability test.

Keywords: Entertainment Systems, Virtual Reality and Interfaces, Multi-Modal Perception
Abstract: This paper describes the improvement of a dropping motion presentation device to enhance the sensation of falling in a VR space. In this study, we developed a bellows actuator-type dropping motion device that is lightweight and bendable in order to improve the sensation of walking when wearing the device. In addition, an exhaust gas control device was fabricated to control the large flow of exhaust air necessary for the bellows actuator to operate, and a two-step dropping motion was realized, which is a motion technique to improve the sensation of falling. In the walking sensation evaluation experiment, the ease of walking when wearing the product and actual walking were evaluated. The results showed that the developed device scored higher than the conventional device in both the subject's subjective sensation and objective evaluation items. There was also a statistically significant difference between these scores, and it was concluded that the improved device contributed to an improvement in the comfort of walking for the wearer.

Keywords: Virtual Reality and Interfaces, Systems for Service/Assistive Applications, Formal Methods in System Integration
Abstract: Recent advancements in deep learning technology have significantly improved the performance of instance segmentation to a practical level. This technology enables the detection, segmentation, and extraction of object regions, offering substantial potential for applications in mixed reality (MR). While most research has focused on detection and segmentation, the application of extraction in realizing MR has received limited attention. In this paper, we propose a framework called ReViSE (Real and Virtual Seamless Editor), which integrates instance segmentation with virtual reality (VR) technology to deliver a wide range of MR experiences. This framework generates diverse MR visuals by applying instance segmentation on original images captured by a camera, and then replacing specified arbitrary object regions with virtual objects. Then, the MR visuals are presented through a head-mounted display (HMD) to provide users with a highly immersive visual experience. Evaluation experiments with a basic implementation show a processing time of 52.67 ms/frame and a display performance of 32.73 fps. The framework has also demonstrated its ability to accurately extract target objects and deliver a high-quality visual experience.

Keywords: Human Interface, Rehabilitation Systems, Virtual Reality and Interfaces
Abstract: Unilateral spatial neglect (USN) is a higher cognitive dysfunction that can occur following a stroke. It is characterized by the impairment in locating, reporting, reacting to, and directing attention toward stimuli presented on the side opposite to the brain lesion. Although it is understood that USN results from the misdirection of attention, the influence of a patient’s attentional state on the symptoms of neglect remains unclear. This study aimed to explore the relationship between attention and neglect by identifying the position and size of the attentional area, as well as the shape characteristics of the field of view (SCFV), using 3D virtual reality (VR). We assessed these characteristics based on the patient’s recognition or non-recognition of displayed objects, reaction time, and gaze behavior within an immersive VR space. The results indicated that the relationship between attention and neglect symptoms varied among patients, suggesting the potential for individualized rehabilitation based on each patient’s attentional state.

Keywords: Virtual Reality and Interfaces, Robotic hands and grasping, Machine Learning
Abstract: Teleoperation is popular due to its several advantages, including the ability to control a robot from a distance and the capacity for the operator to manage the robot safely. However, teleoperation also presents challenges, including operational complexity and the requirement for a certain level of proficiency from the operator. For instance, when attempting to grasp an object via teleoperation, issues such as communication delays, inadequate feedback from the robot to the operator, and the complexity of the grasping trajectory can arise. To address this issue, we propose an intuitive teleoperation method that facilitates data collection using VR devices and a technique for generating object-grasping motions through predictive learning with the collected data. First, we collect the motion data while the robot is teleoperated using a VR device. The collected motion data is used to create a predictive model through predictive learning, which in turn is used to generate object-grasping motions. This approach allows us to collect motion data suitable for machine learning while performing intuitive teleoperation. It also enables the generation of object-grasping motions with simple operations, making robot teleoperation experience similar to a VR game. We evaluated our approach's ability to generate object-grasping motion with predictive model. The results show that our approach can generate object-grasping motions with a certain level of success. In light of our results, we discussed the factors that pose challenges to predictive learning and explored the future prospects of this approach.

Keywords: Systems for Service/Assistive Applications, Human-Robot/System Interaction, Control Theory and Technology
Abstract: Telesport, which involves playing sports via avatar robots, has the potential to provide people with physical limitations with the chance to participate in sports, as it allows them to replace their bodies with robots. However, the delay in the teleoperation system makes real-time operation difficult, and it is challenging to operate the agile robot as intended. In this study, we focused on overhand throwing and treated the problem of it being difficult to throw the ball in the intended direction and speed using manual control. In order to accurately realise the agile movements that a user intends, we propose an assistance system that intervenes with automatic control based on the estimated future user's intent for manual control. Furthermore, this assistance system blends manual and automatic control seamlessly to prevent the user from feeling disconnected from the robot due to the intervention of automatic control. The assistance system was evaluated by measuring the direction and speed of the ball thrown overhand, and by assessing whether the user's intent was reflected. As a result, by making the assistance system effective, manual and automatic control were seamlessly blended, and it was confirmed that the throwing motion intended by the user was accurately reflected in the robot.

Keywords: Haptics and tactile sensors, Robotic hands and grasping, Multi-Modal Perception
Abstract: This study addresses the challenge of limited dexterity in teleoperation tasks caused by the absence of sensory feedback and visual occlusions. Our approach anticipates interactions by virtually enlarging the robot's gripper and calculating the overlapping volume with nearby objects. To intuitively convey this information to the operator in real time, we employ haptic feedback by adjusting the stiffness of the controller's buttons based on the proximity of the detected objects. The system has been tested in a simulation environment with the aim of achieving a good position to grasp a target. In situations where the target was not fully grasped, operators using haptic feedback reached the "Best" position 28% faster than those without it, indicating enhanced situational awareness and control. The percentage of the target inside the gripper was notably higher, and centering and alignment errors were minimized, suggesting more precise grasping with less damage to the robot or the object. Although the collision frequency remained similar, the severity of collisions—both maximum and average—was reduced when haptic feedback was employed. In situations where the target was fully grasped, there are minor differences when performing with or without haptic feedback. These findings suggest that haptic feedback improves the user's awareness of the robot’s interactions with its environment and enhance the operator’s ability to make informed decisions.

Keywords: Virtual Reality and Interfaces, Human-Robot/System Interaction
Abstract: Planning efficient motions of robots is often inhibited by the difficulty to spatially imagine and enhance their dexterity and agility in the task space. To accommodate this complexity, meet short changeover times, and support inclusion along with sustainability goals, robot operators with different experiences need responsive and actionable interfaces. These enable the prediction and optimization of hidden (i.e., internal) performance-critical properties of physical robots, such as their velocity manipulability in the task space and thermal loads, directly on the concerned components. We address these challenges by developing a virtual spatial augmentation of physical robots that translates their complex, otherwise invisible manipulability and thermal states, into accessible and interactive visual cues easily interpreted and used by even novices to improve the robot dexterity in the null-space and anticipate issues due to overheating joints. An upskilling immersion based upon augmented reality and enriched with overlaid digital twins is leveraged to this end. While following values targeted by our overarching Metarobotics framework, we stress the non-invasive and sustainable characteristic of the approach. Furthermore, our framework transparently embeds in existing robotized industrial settings, systems of systems, and workflows without production standstills. We emphasize on its semantic interoperability, versatility, and openness driven by the OPC UA standard and share results from practical experiments.

Keywords: Human-Robot Cooperation/Collaboration, Human Interface, Machine Learning
Abstract: The body integration system in which multiple users co-operate a single-robot avatar improves operability. However, collaboration among users is essential for smooth operation. In this study, we proposed bidirectional operation predictions for the actions of each operator with their partner in the body integration system. The robot arm and gripper are controlled by two operators. At the same time, based on each operator's actions, the system predicts the operations 0.3 seconds ahead using machine learning, and the prediction results are visually presented to the other operator. To confirm the operability of the system and its effect on cooperation between operators, we conducted pick-and-place experiments under conditions with and without bidirectional prediction. The results of the subjective evaluation and operation performance suggested that the subjective rating in smooth cooperation was significantly improved, and using the prediction could improve the similarity in the operations of two users who provided a small mental workload and shortened psychological distance.

Keywords: Automation Systems, Environment Monitoring and Management, Machine Learning
Abstract: Morning glories prevent soybean growth and reduce yields. However, early morning glories are small and difficult to detect visually in the vast fields. Semantic segmentation effectively estimates the location of weeds using images captured by drones, but a large amount of image data is required to ensure high prediction accuracy and an open dataset of plants with a wide variety of species is limited. In this research, we propose an image generation system of pseudo-dense morning glory using PGGAN to increase the volume of the dataset for training. The proposed pseudo images can configure the density of multiple morning glory using the distance. As a result of including these pseudo-densely morning glories, the F_2 score of the estimation was 0.404.

Keywords: Systems for Field Applications, Automation Systems, Sensor Fusion
Abstract: Continuous crop monitoring is essential for inspection of pests, diseases, and the evaluation of crop growth. It requires registering sensor data from the same field over a period of time. Agricultural fields can be effectively monitored by cameras or LiDAR mounted on unmanned aerial vehicles (UAVs). However, due to significant changes in crop growth and soil conditions over different measurement periods, conventional data registration methods often fail to reduce accuracy reduction.

In this study, we proposed a point cloud registration method that utilizes static geometric information of crop rows and terrain to align maps from different measurement periods. In our experiments, we first created a map with accurate positional data using LiDAR at the beginning of the season. We then generated maps by Structure from Motion (SfM) 5 days later, when ground information had notably changed due to tractor activity, and the other map 19 days later when crops had grown considerably. The proposed method was applied to register the maps generated in different periods. The result of a demonstration by using precise positional information obtained at the start of the season showed that we were able to align maps taken up to 19 days apart with displacements of no more than 30 cm.

Keywords: Machine Learning, Vision Systems, Systems for Field Applications
Abstract: Automation in agriculture is increasingly critical for addressing global food security and sustainability challenges. This paper presents a novel crop detection method using relative positional relationships, specifically designed for small weeding robots. Unlike traditional approaches that rely heavily on visual characteristics and large annotated datasets, our method leverages the spatial arrangement of plants to distinguish crops from weeds, thereby reducing the dependency on extensive data collection and annotation efforts. We implemented a multi-stage detection system that first identifies all plants using an object detection algorithm and then classifies them based on their positional and size information. Experimental results on soybean datasets demonstrate that our approach achieves AP of 71.3% for soy crops and 27.2% for weeds in environments not included in the training dataset, showing comparable effectiveness to traditional visual-based detection methods in scenarios with limited data. This advancement offers potential for enhancing the adaptability and efficiency of agricultural automation technologies.

Keywords: Systems for Field Applications, Systems for Service/Assistive Applications, Vision Systems
Abstract: The strawberry industry faces significant challenges such as harvest inefficiency and inconsistent quality due to labor shortages and reliance on inexperienced workers. One factor contributing to this is the lack of quantified criteria for strawberry harvesting decisions. In this study, we propose a harvest support system that uses smart glasses as a medium to provide real-time feedback on quantified ripeness and size information through computer vision technology. In the experiment, we compared the efficiency and accuracy of harvesting between cases where humans made harvesting decisions alone and cases where they were supported by the developed system to verify the usefulness of the developed system. The results showed that the system improved the harvesting speed by 17%, the accuracy of size evaluation by 25% and ripeness evaluation by 8% compared to the case where only human judgment was used. These results suggest that this system not only improves productivity and ensures consistent quality but also is expected to increase the number of new entrants in the strawberry industry, potentially addressing the labor shortage in the strawberry sector.

Keywords: Human-Robot Cooperation/Collaboration, Vision Systems, Systems for Field Applications
Abstract: This study proposes an innovative approach for tracking agricultural workers in narrow furrows, aimed at enhancing the performance of agricultural support robots. The method integrates RGB-D camera-based skeleton extraction with a Space-Time-Separable Graph Convolutional Network (STS-GCN) for lower limb motion prediction, and introduces a novel fusion algorithm that combines these predictions with real-time observations. Experimental results demonstrate the proposed method's ability to maintain accurate worker tracking even in occluded scenarios by complementing observations with predictions. This research provides insights into the effectiveness of combining skeletal graph-based motion prediction with real-time observations for robust worker tracking in challenging agricultural environments.

Keywords: Intelligent and Flexible Manufacturing, Sensor Fusion, Motion and Path Planning
Abstract: The universal task of cutting requires the extensive use of knives and other cutting tools in our daily lives. When cutting soft and hard objects such as meat, fish, and fruit, humans can cut efficiently without damage by responding to changes in the food with adaptive movements. The effect of knife manipulation on a robot is imperative. This research aims to develop a cutting motion system capable of handling various foodstuffs and automating cooking operations in restaurants and food manufacturing plants by studying human sensory knife manipulation and its impact on the reaction force while cutting foodstuffs. In order to automate the cooking process for various types of food ingredients, it is necessary to recognize the shape of the targets and determine the position and angle of the knife accordingly. In this study, we developed a method that can take into account the shape and flexibility of the food in order to suppress deformation. This research measured the reaction forces when cutting various food materials and developed a slice-cutting motion system with a trajectory that is appropriate for the shape of the food material. This paper describes the slice cutting system and the results of a demonstration experiment.

Keywords: Environment Monitoring and Management, Path Planning for Multiple Mobile Robots or Agents, Automation Systems
Abstract: This paper addresses the problem of task division based on 3D urban models for visual inspection using a UAV with multiple flights. When inspecting large structures for damage detection, UAVs are often flown multiple times to obtain high-resolution data from all areas. Research is progressing on coverage path planning (CPP) methods for data collection in a single flight, and the 3D models of the target required for this can be readily obtained. However, the automation of task division for each flight has not been extensively studied. Because the scale of the target is large, it is necessary to execute a task division method that optimizes the performance efficiency of UAVs within a practical calculation time. This paper presents a task division method for data collection by multiple flights of a UAV based on the decomposition of the 3D urban models. Forming a framework in which data collection tasks are divided based on the 3D mesh decomposition, and then the CPP method is applied to each task, this enables efficient inspection of large-scale structures. In this paper, we implement three basic methods based on the strategies of avoiding going far, equalizing the amount of each task, and reducing unnecessary movement as preliminary research. Each method applies to objects on a scale of several hundred meters, and evaluates their performance in automated data collection.

Keywords: Environment Monitoring and Management, Systems for Search and Rescue Applications, Systems for Field Applications
Abstract: This study proposes a novel method for estimating radiation source distribution in a reactor pressure vessel (RPV), especially focusing on the fuel debris retrieval at the Fukushima Daiichi Nuclear Power Plant. Before the fuel debris retrieval, it is necessary to grasp the internal situation of a primary containment vessel (PCV). Our research group is planning to insert an investigation unit from the top of the PCV and measure radiation using non-directional gamma-ray detectors. The proposed method defines the bottom of the RPV as two-dimensional grids and estimates the distribution of radiation sources. The proposed method consists of three steps: database construction, choosing candidates distributions from the dataset and comparing them with the prior information, and additional measurements. Simulation experiments demonstrate that the proposed method is possible to successfully estimate the radiation sources distribution in the RPV with a limited number of measurements.

Keywords: Vision Systems, Robotic hands and grasping, Soft Robotics
Abstract: Soft robotic grippers are highly adaptable to various objects because they can deform and fit object shapes. However, grasping stability may change owing to the posture of the gripper while grasping an object. For a stable grasp, it is necessary to estimate the grasping posture before the grasp, namely pre-touch estimation. In particular, for soft robotic grippers, an important factor in the grasping posture is gripper deformation. The deformation of the gripper depends on the intrinsic characteristics of the gripper deformation (e.g., stiffness) and the contact positions between the gripper and object, that is, how the gripper can deform and where on the gripper is in contact with the object. Deformation characteristics vary from one gripper to another, and the contact positions change according to the characteristics, gripper location, and object shape. Thus, an estimation method that considers these conditions is required to achieve a pre-touch estimation of the deformation of soft robotic grippers. This study presents a vision-based method for estimating the deformation of a soft robotic gripper prior to grasping an object. The entire method is performed before the gripper grasps an object. First, the deformation model that shows the manner in which the gripper can deform is defined using three approaches: discretization of the gripper based on a model of a serial chain of rigid bodies connected with a spring joint, the bending angle of the entire gripper, and piecewise constant curvature. Next, using an image, the bending angle of the entire gripper is acquired to calibrate the deformation model. Then, the contact points between the gripper and object are predicted by obtaining their contours from an image. Finally, the deformation of the entire gripper is estimated based on the deformation model and predicted contact points. Three experiments were conducted to evaluate the accuracy and versatility of the proposed method.

Keywords: Environment Monitoring and Management, Building Automation, Autonomous Vehicle Navigation
Abstract: In this paper, a novel framework for real-time localization through pre-built map updates by 3D LiDAR mounted on a robot is proposed. Autonomous mobile robots usually use high-precision 3D maps (i.e., HD maps) built in advance. However, as time passes, the environmental map changes from the actual environment, adversely affecting autonomous navigation. In addition, it is costly to recreate an HD map every time the environment changes. Therefore, it is necessary to update environmental maps simply and frequently. We propose a method for real-time map updating in the form of point clouds and validate the results of integrating map updates with localization.

Keywords: Human-Robot/System Interaction, Human-Robot Cooperation/Collaboration
Abstract: We deal with a visualisable and adjustable gesture-based system for operating home appliances with gestures that can be easily operated and adjusted by anyone. The system uses “Command Spaces” in which commands for home appliance operation are tied to a space. These command spaces are constructed using body-relative coordinates using gestures at the start of the operation, which enables operation regardless of the posture and location of the operator. Different manners people are comfortable performing gestures in different postures. Gestures vary from person-to-person. In this paper, we construct a system that visualizes them using Mixed Reality via the Microsoft HoloLens2. This allows the operator to freely visualize and adjust them, improving the usability of the appliance operation system. Comparative experiments showed that operation time, operation accuracy, and usability were improved when the system was used. This shows that the system facilitates learning about home appliance operation.

Keywords: Vision Systems, Autonomous Vehicle Navigation, Motion and Path Planning
Abstract: This paper deals with 6-DOF (six-degrees of freedom) SLAM (simultaneous localization and mapping) using general RGB cameras in extremely dark environments, assuming lunar and planetary environments. One of the challenging points is to obtain traceable features in dark environments. To cope with such problems, we proposes a new landmark design using phosphorescent materials that emit light without using any power source and its recognition method. We demonstrated that the performance of 6-DOF SLAM can be improved by our new SLAM framework with new landmark information in an uneven terrain where the odometry of a mobile robot is inaccurate.

Keywords: Machine Learning, Vision Systems, Environment Monitoring and Management
Abstract: This paper describes a 3D reconstruction based on grouping similar structures for the aim of generating 3D information for understanding the workspace from images acquired inside the Primary Containment Vessel (PCV) of the Fukushima Daiichi Nuclear Power Station. In the decommissioning works, preliminary surveys are carried out in the PCV, and the workers need to understand the workspace from a large number of camera images, which requires a great deal of effort. We are currently working on 3D reconstruction from PCV camera images; however, one of the challenges is to improve the visibility of the reconstructed model containing noise and artifacts. In this study, we propose a method of grouping similar structures on images and utilizing predicted group labels for 3D reconstruction process to highlight structures shapes and to refine 3D modeling. Our key idea is to perform unsupervised segmentation for grouping similar structures that are suitable for images acquired in the PCV because they are difficult to assign correct semantics for unclear structures and the few learning resources. We show on the reasonable performance the proposed method by validating it using video images of a typical plant environment and survey videos of the PCV taken under adverse conditions, such as radiation noise.

Keywords: Logistics Systems, Automation Systems, Vision Systems
Abstract: The logistics industry is facing major labor shortages owing to the increasing production volume driven by the continued expansion of e-commerce. This situation has accelerated the development of solutions such as autonomous forklifts. For these forklifts to perform stable material handling, the accurate detection of the load state is essential. However, logistics data often contain sensitive information related to customer products, hindering the collection of comprehensive datasets for developing detection technologies based on machine learning. We propose a method for accurately detecting the position and shape of a load as a mask using limited load data and subsequently identifying the interference state between loads based on the predicted masks. The proposed method lever-ages instance segmentation pre-trained with formula-driven supervised learning (FDSL) to achieve highly accurate mask prediction, even with limited labeled data for fine-tuning. Pre-training using FDSL leads to a high detection accuracy with a mean average precision (intersection-over-union threshold of 90) of 91.0% using only 400 images. Furthermore, interference detection based on the predicted masks reaches high rates, with a precision of 95.0% and recall of 95.0% on an evaluation set that includes loads with and without interference. Our findings indicate that accurate load interference detection can be achieved with limited labeled data, possibly contributing to the advancement of automation in the logistics industry.

Keywords: Mechanism Design, Robotic hands and grasping, Mechatronics Systems
Abstract: Cabling tasks (pulling, clipping, and plug insertion) are today mostly manual work, limiting the cost-effectiveness of electrification. Feasibility for the robotic grasping and insertion of plugs, as well as the manipulation of cables, have been shown in research settings. However, in many industrial tasks the complete process from picking, insertion, routing, and validation must be solved with one system. This often means the cable must be directly manipulated for routing, and the plug must be manipulated for insertion, often in cluttered environments with tight space constraints.

Here we introduce an analysis of the complete industrial cabling tasks and demonstrate a solution from grasp, plug insertion, clipping, and final plug insertion. Industrial requirements are summarized, considering the space limitations, tolerances, and possible ways that the cabling process can be integrated into the production process.

This paper proposes gripper designs and general robotic assembly methods for the widely used FASTON and a cubical industrial connector. The proposed methods cover the cable gripping, handling, routing, and inserting processes of the connector. Customized grippers are designed to ensure the reliable gripping of the plugs and the pulling and manipulation of the cable segments. A passive component to correct the cable orientation is proposed, allowing the robot to re-grip the plug before insertion. In general, the proposed method can perform cable assembly with mere position control, foregoing complex control approaches.

This solution is demonstrated with an industrial product with realistic space requirements and tolerances, identifying difficult aspects of current cabling scenarios and potential to improve the automation-friendliness in the product design.

Keywords: Motion and Path Planning, Systems for Service/Assistive Applications
Abstract: This paper describes a method for solving task planning and motion planning problems simultaneously. We target a fetch-and-carry of a small item by a single-arm mobile manipulator and introduce a method that can generate a sequence of actions and motions required for each action, even in environments with narrow open spaces such as corridors. In addition, to deal with a case where another object is already placed at the target location, we introduce a push-aside action and extend the previous method to include this action. As our method is formulated using Mixed Integer Linear Programming (MILP), the calculation time is relatively short, irrespective of the complexity of the target problem. To verify the efficiency of the proposed method, we performed a quantitative evaluation through simulation and conducted experiments on an actual mobile manipulator to verify feasibility of the methods.

Keywords: Intelligent and Flexible Manufacturing, Motion and Path Planning, Automation Systems
Abstract: We propose an assembly system involving two robotic arms with flexible mechanisms.These flexible joints provide passive compliance, allowing the robots to absorb positional errors between assembly parts and their respective destinations.Our machine learning-based object detection system shows robustness against variations in color, shape, and ambient lighting, achieving high positional accuracy.This system achieves the execution of diverse assembly tasks using a single setup, facilitated by a universal gripper and machine learning capabilities. Furthermore, the flexible joints minimize the need for precise trajectory planning.In our study, we implemented both an instruction support system and a position error correction system using object detection. We conducted real-world assembly experiments to evaluate these systems, focusing on tasks such as screw tightening and belt drive assembly, with alignment achieved through vision feedback and force feedback control.

Keywords: Automation Systems, Robotic hands and grasping
Abstract: In this study, we propose an exploratory binpicking system with less installation space than a human worker. The robot adjusts the arm tip position based on quasistatic deformations of the compliant fingers. It also estimates the number of grasped objects by frequency analysis of the dynamic deformations. The proposed system has two advantages because it performs bin-picking using only deformations. The first is that all control can be performed on small on-board computer board (Raspberry Pi 4B). Second, there is no need to place and mount a 3D vision sensor. The control system can also be reconfigured without any learning time when shape of the picked objects changes. In bin-picking experiments, we confirmed that our system achieved a picking success rate of over 85% and a bin picking tact time within 30 s for two types of metal bolts.

Keywords: Control Theory and Technology, Intelligent and Flexible Manufacturing
Abstract: Many robotic tasks, such as polishing or grinding, involve maintaining contact with and applying a force against the environment while following a given trajectory. In this paper, we present an adaptive admittance controller that aligns its control parameters online to be in the direction of an estimate of the surface normal vector. This essentially allows a robot to follow an unknown surface, as is the case in uncalibrated setups or quick changeover production. We present and compare three different surface normal estimation algorithms: the integral adaptive law and two Riemannian manifold based algorithms.

Our experimental results show that the adaptive controller using the simple Riemannian gradient descent yields the lowest tracking error of the three. It has 73 % decrease in positional error and 43 % decrease in angular error compared with the controller with the integral adaptive law, and overall is effective at aligning the robot tool online against surface moving in an a priori unknown pattern.

Keywords: System Simulation, Factory Automation, Automation Systems
Abstract: Considering energy consumption and dynamical behavior within the simulation of manufacturing systems emerges as important objective for their development, commissioning, and operation. Currently, these aspects are often neglected in many fields including virtual commissioning. This work presents an approach to extend highly detailed virtual commissioning models of components from industrial automation by dynamics models for the calculation of energy consumption in an offline simulation and for the reuse in an online simulation. To validate the presented approach, an exemplary offline virtual commissioning simulation of a pick-and-place robot application is implemented and subsequently transferred to an online simulation. The validation shows a good accuracy of the simulated energy consumption, enabling virtual commissioning engineers to analyze and optimize energy consumption already in the design phase of manufacturing systems. Additionally, the dynamical behavior was well reflected in the online simulation, contributing to the capability of simulation in the operation phase.

Keywords: Formal Methods in System Integration, Factory Automation, Intelligent and Flexible Manufacturing
Abstract: This paper explores the integration of Open-set Recognition (OSR) into intelligent manufacturing systems, specifically for CNC machines. The authors propose a novel hardware-accelerated OSR model using a Field-Programmable Gate Array (FPGA) for real-time classification of machined surface quality. This approach improves performance by adapting to unknown conditions, which is crucial in dynamic manufacturing environments. The hardware implementation provides significant speed advantages over traditional software models, reducing downtime and enhancing productivity by automating machine setup and feedback processes. Experimental results demonstrate the effectiveness of the model in surface smoothness classification, outperforming traditional random forest models.

Keywords: Systems for Field Applications, Automation Systems, Mechatronics Systems
Abstract: To automate the process of breaking up a building into small pieces, it is necessary to discriminate rebars from concrete lumps to remove rebars. This paper proposes a method for discriminating rebar from concrete lumps using Gabor filters. Our discrimination system uses a filter bank consisting of multiple Gabor filters. Using the created filter bank, feature extraction is performed from each segmented region, and the image is classified into rebar and concrete chunks using a support vector machine. The system was run on images of actual rebar and concrete lumps and achieved a discrimination accuracy of 89.4%.

Keywords: Intelligent Transportation Systems, Sensor Fusion, Environment Monitoring and Management
Abstract: A system that can automatically determine where and when the prior knowledge map needs to be updated according to the new observation would be helpful in avoiding poor accuracy and reliability of map matching for self-position estimation. Such a system is also useful for road maintenance and management to identify areas where repainting work is required. Therefore, as a first effort to develop such a system, this paper proposes a lidar-camera fusion framework to extract road marking areas from reflectance maps for quantifying the reflectance of the areas, which can be detected and confirmed using camera images.

Keywords: Sensor Fusion, Systems for Field Applications
Abstract: We propose a high-precision odometry method that fuses visual-lidar odometry with orientation information estimated by ceiling line features, aiming to achieve millimeter-level localization required at construction sites. Visual-Lidar Odometry and Mapping has demonstrated top-class accuracy with an error of 50 mm in benchmarks. However, when evaluated in environments with large open walls, such as indoor construction sites with few shape and image features, the method was found to be less accurate primarily due to errors in the orientation estimation. To address this issue, we propose a method that combines the use of ceiling line features to estimate the robot's orientation angle, thereby improving the accuracy of the odometry position estimates. We evaluated the accuracy using a real robot in an environment similar to a construction site and achieved an average error of 6.2 mm (straight-line route), 7.6 mm (rectangular route), surpassing the accuracy of conventional methods. These results confirm the effectiveness of utilizing orientation correction from ceiling line features.

Keywords: Integration Platform, Multi-Robot Systems, Systems for Field Applications
Abstract: On the lunar surface, modular robots that can perform multiple tasks through coupling, decoupling, and reconfiguration are expected to play a significant role due to considerations of cost, work efficiency, and fault tolerance. One of the tasks on the lunar surface is the construction of high-rise solar power towers to secure power sources. This study proposes a construction method using modular robots to build high-rise structures on the lunar surface. The shape of both the robots and the structural modules is designed as truncated octahedrons, and four prototypes of these modules have been developed in this study. Additionally, we have previously proposed a method for determining the configuration, position, and posture of modules by discretizing space into truncated octahedron shapes, focusing on their space-filling properties. In this paper, we propose a construction method for building high-rise structures using both the previously prototyped modules and newly propose modules. Significant points of these proposing concept is based on discretizing the workspace using three-dimensional shapes and implements methods of an automated system for the desired task, which can be said that the proposed methods form an efficient and coherent system integration platform on the target missions.

Keywords: Control Theory and Technology, Building Automation, Multi-Robot Systems
Abstract: Since the successful Apollo program, humanity is once again aiming to return to the Moon for scientific discovery, resource mining, and inhabitation. Upcoming decades focus on building a lunar outpost, with robotic systems playing a crucial role to safely and efficiently establish essential infrastructure such as solar power generating towers. Similar to the construction of the International Space Station (ISS), shipping necessary components via modules and assembling them in situ should be a practical scenario. In this context, this paper focuses on the integration of vision, control, and hardware systems within an autonomous sequence for a dual-arm robot system. We explore a perception and control pipeline specifically designed for assembling solar panel modules, one of the benchmark tasks. Ad hoc hardware was designed and tested in real-world experiments. A mock-up of modular solar panels and active-passive connectors are employed, with the control of this grappling fixture integrated into the proposed pipeline. The successful implementation of our method demonstrates that the two robot manipulators can effectively connect arbitrarily placed panels, highlighting the seamless integration of vision, control, and hardware systems in complex space applications.

Keywords: Systems for Field Applications, Machine Learning
Abstract: In recent developments, autonomous racing has garnered attention as it aims to overcome the limitations of standard autonomous driving systems. Achieving safe racing conditions necessitates both fast and long-range perception. However, current 3D LiDAR object detection methods face challenges with high computational costs and limited detection ranges. These issues make them unsuitable for racing scenarios. To address these challenges, this paper proposes a clustering-based long-range vehicle detection method that relies solely on LiDAR. First, the method removes ground points and foreground points and clusters the remaining points. Subsequently, these clusters are classified as vehicles using AdaBoost, generating 2D bounding boxes in the range view. Experimental results demonstrate superior performance, achieving a computational efficiency of 53 Hz and a long-range detection accuracy of over 80%, compared to voxel-based and range-based methods. This approach offers a viable solution for autonomous racing environments.

Keywords: Network Systems, Software Platform, Hardware Platform
Abstract: Seamless and reliable communication is crucial across personal and industrial domains in today's interconnected world, ranging from multimedia streaming and augmented/virtual reality to smart manufacturing. This need extends to wireless communication, which supports mobile applications and offers greater flexibility. Time-Sensitive Networking (TSN) addresses stringent real-time demands with low latency, low jitter, and high availability, relying on precise time synchronization via the Precision Time Protocol (PTP). While PTP is widely adopted in wired networks, its usage in wireless networks is less common. This paper investigates the potential of Ultra-Wide Band (UWB) technology to achieve high-precision clock synchronization in Wireless-TSN (W-TSN). Traditionally used for indoor localization owing to its high-precision timestamping, UWB offers reliability and resilience against multipath fading. This research integrates UWB into Openwifi, an open-source Software Defined Radio (SDR) Wi-Fi chip design augmented with TSN capabilities, serving as the W-TSN solution. Significant improvements in synchronization accuracy were achieved by increasing the timestamping resolution and refining the clock drift correction algorithm within the Openwifi FPGA. This resulted in a Mean Absolute Error (MAE) of 13.1 ns and a 90th percentile (P_90) clock error of 25.0 ns. The integration of UWB technology further improved clock synchronization accuracy by roughly 25%, achieving an MAE of 9.98 ns with a P_90 error of 20.0 ns. Finally, the study highlights UWB-based synchronization's potential applications and benefits, including inherent localization capabilities, while noting limitations such as range constraints and added complexity.

Keywords: Systems for Field Applications, Mechatronics Systems, Automation Systems
Abstract: The number of port facilities in Japan that were constructed more than 50 years ago is rapidly increasing. Therefore, it is necessary to take measures to ensure the safety and maintain the functionality of the facilities. However, in conventional inspections, inspectors board a boat and use a camera to take pictures of cracks, peeling and other defects on the wall surface to analyze the degree of deterioration, which requires a huge amount of time and money. Therefore, in this study, we have developed an unmanned surface vehicle for efficient inspection of the underside of piers. This paper proposes a sway suppression control and a path following control, and reports the results of autonomous navigation experiments using the robot in a real ocean environment.

Keywords: Multi-Robot Systems, Path Planning for Multiple Mobile Robots or Agents, Autonomous Vehicle Navigation
Abstract: Multi-robot autonomous exploration in an unknown environment is an important application in robotics. Traditional exploration methods only use information around frontier points or viewpoints, ignoring spatial information of unknown areas.Moreover, finding the exact optimal solution for multi-robot task allocation is NP-hard, resulting in significant computational time consumption. To address these issues, we present a hierarchical multi-robot exploration framework using a new modeling method called RegionGraph. The proposed approach makes two main contributions: 1) A new modeling method for unexplored areas that preserves their spatial information across the entire space in a weighted graph called RegionGraph. 2) A hierarchical multi-robot exploration framework that decomposes the global exploration task into smaller subtasks, reducing the frequency of global planning and enabling asynchronous exploration. The proposed method is validated through both simulation and real-world experiments, demonstrating a 20% improvement in efficiency compared to existing methods.

Keywords: Biologically-Inspired Robotic Systems, Decision Making Systems, Systems for Search and Rescue Applications
Abstract: In high-stress situations, guiding a non-communicating, location-unaware swarm to safe area is challenging, especially as they self-organize into clusters with nearby agents. This paper presents a novel decentralized non-interacting swarm algorithm to form guided swarm clusters autonomously in an area without nearby danger, termed as "safe area". The proposed strategy guides the swarm of robots towards safe area to form self-clusters in a bounded environment. It is achieved by fixing the locations of static obstacles which prevents cluster formation in the danger area. We evaluate the performance based on cluster formation in the safe area. The Monte-Carlo simulations are performed with varying percentage occupancy of obstacles in danger area, where we achieved percentage of convergence (PoC) in safe area over 85% (on an average). The proposed algorithm potentially find its application in a crowded environment, while evacuating to a safer area during emergency.

Keywords: Autonomous Vehicle Navigation, Multi-Robot Systems
Abstract: This study proposes to employ a rendering engine as a tool for recognition and control. The example of the study primarily demonstrates controlling two types of vehicles: a rover and a drone. The controls require recognizing a three-dimensional space and simulating movement in advance, and a renderer embedded in the controller enables more intuitive recognition and control. For example, it is possible to match the geometry and camera images obtained from the rover by switching them to the drone's point of view. The renderer is newly implemented and embedded in the controller to avoid the heavy computational demands of commercial engines. Consequently, this paper presents the corresponding experiments in the experimental section, with explanations based on simulations discussed in the implementation section. In conclusion, this study proposes a renderer that has been thoroughly researched as a recognition tool and shows an example.

Keywords: Control Theory and Technology, Multi-Robot Systems
Abstract: This paper deals with cooperative wind disturbance estimation by multiple quadrotor drones. The authors have developed a cooperative wind disturbance estimation method combining the optimal control and adaptive control. However, only disturbances affecting translational motion have been considered and torque disturbances influencing rotational motion have not been taken into account. This paper addresses a cooperative disturbance estimation problem in a more realistic situation where torque disturbances exist as well. The two types of disturbances are estimated and compensated simultaneously. A disturbance estimator is designed based on the Lyapunov stability theory. Numerical simulations show the effectiveness of the proposed method and also indicate the necessity of simultaneous estimation of translational disturbances and torque disturbances.

Keywords: Multi-Robot Systems, Intelligent Transportation Systems
Abstract: Multi-object transport using multi-robot systems has the potential for diverse practical applications such as delivery services owing to its efficient individual and scalable cooperative transport. However, allocating transportation tasks of objects with unknown weights remains challenging. Moreover, the presence of infeasible tasks (untransportable objects) can lead to robot stoppage (deadlock). This paper proposes a framework for dynamic task allocation that involves storing task experiences for each task in a scalable manner with respect to the number of robots. First, these experiences are broadcasted from the cloud server to the entire robot system. Subsequently, each robot learns the exclusion levels for each task based on those task experiences, enabling it to exclude infeasible tasks and reset its task priorities. Finally, individual transportation, cooperative transportation, and the temporary exclusion of tasks considered infeasible are achieved. The scalability and versatility of the proposed method were confirmed through numerical experiments with an increased number of robots and objects, including unlearned weight objects. The effectiveness of the temporary deadlock avoidance was also confirmed by introducing additional robots within an episode. The proposed method enables the implementation of task allocation strategies that are feasible for different numbers of robots and various transport tasks without prior consideration of feasibility.

Keywords: Multi-Robot Systems, Intelligent Transportation Systems
Abstract: Multi-robot cooperative transport is crucial in logistics, housekeeping, and disaster response. However, it poses significant challenges in environments where objects of various weights are mixed and the number of robots and objects varies. This paper presents Task-priority Intermediated Hierarchical Distributed Policies (TIHDP), a multi-agent Reinforcement Learning (RL) framework that addresses these challenges through a hierarchical policy structure. TIHDP consists of three layers: task allocation policy (higher layer), dynamic task priority (intermediate layer), and robot control policy (lower layer). Whereas the dynamic task priority layer can manipulate the priority of any object to be transported by receiving global object information and communicating with other robots, the task allocation and robot control policies are restricted by local observations/actions so that they are not affected by changes in the number of objects and robots. Through simulations and real-robot demonstrations, TIHDP shows promising adaptability and performance of the learned multi-robot cooperative transport, even in environments with varying numbers of robots and objects.

Keywords: Multi-Robot Systems, Mechanism Design, Mechatronics Systems
Abstract: Robots are used for various industries and must adapt to various environments. For this purpose, machine learning has been extensively studied as a method to output motion adapted to the environment. However, if the robot's hardware is not adaptable to the environment, the range of the robot's motion using machine learning is limited. Therefore, this paper focuses on self-reconfigurable robots that change their own shape to adapt to various environments. Most self-reconfigurable robots are assembled with connectors oriented facing in a specific direction; however, their assembly is time-consuming because of the required alignment of the connectors. Therefore, this paper presents an omni-directional connector for self-reconfigurable robots and proposes a fast assembly.

Keywords: Motion and Path Planning, Robotic hands and grasping, Machine Learning
Abstract: In this paper, we focus on object storage by a dual-arm robot. The robot is required place objects grasped by its hands onto a shelf. The task is treated in terms of motion planning. For image inputs captured by a camera, the object storage motion outputs are derived by the robot itself based on a motion planner. For this purpose, we propose a motion planner based on Convolutional Neural Network, CNN. In order for the robot to plan the storage motion toward the shelf composed of multiple spaces, goal directions for both hands, indicating the center coordinates of target spaces, are used as inputs in addition to images and current hand positions. For these multimodal inputs into the motion planner, shortcut connections are further applied to convolutional layers in the CNN to adjust learning bias. Through the experiments, we discuss the effectiveness of the motion planner with the goal directions and shortcut connections for object storage by the robot.

Keywords: Human-Robot Cooperation/Collaboration, Environment Monitoring and Management, Human-Robot/System Interaction
Abstract: The objective of this research is to construct an intelligent human-robot environment that can infer human behavorial intentions and adjust the space accordingly. In this research, we perform preliminary studies and verify whether inferring of human behavorial intention can be done from image information alone. First, the vision and Language Model (VLM) and object detection methods are used to infer possible human actions for each object detected in images. Differences between inference results and actual behavior are identified and methods needed for more accurate inference are discussed. The spatial relationship between the skeletal points and the object by observation reveals which skeletal points to focus on in order to predict the behavior. We confirmed that it is possible to predict behaviors by focusing on the neck point for actions performed with the clear intention of sitting on or passing by a chair. Parameters for the neck skeletal points are selected and each behavior is predicted by a Temporal Convolutional Network (TCN) with 91% performance. Through preliminary experiments, we discuss the methods necessary for inferring human behavioral intentions from images.

Keywords: Vision Systems, Machine Learning
Abstract: In this study, we investigate the optimal capture conditions and features for recognizing frozen road surface conditions using polarization images. We focus on three types of road surfaces— asphalt, concrete, and metal—and classify them into dry, wet, and frozen states. Among the capture conditions, in particular, we examine how the positional relationship between the camera and the lighting influences the recognition of frozen road surface conditions. The results indicated that when the camera and lighting were aligned in the same direction, frozen conditions were successfully recognized on the asphalt surface. Conversely, when the camera and lighting were positioned in opposite directions, frozen conditions were observed on the concrete and metal surfaces. To evaluate the features used in image processing, we used a support vector machine to classify road surface images based on color and polarization information. The results demonstrate successful classification of images into frozen and wet/dry states. In this study, we clarified the appropriate capture conditions and features for frozen condition recognition based on the relationship between frozen road surface conditions and polarization characteristics, as well as the classification outcomes using these features.

Keywords: Vision Systems, Machine Learning, Sensor Fusion
Abstract: Most wire rope inspection work for construction machinery is performed manually at construction sites. Therefore, there are restrictions on the work environment, such as the equipment that can be used for inspection and the working time, and as a result, inspection results depend on the skill of the inspector. The purpose of this research is to develop a wire rope inspection system that is not affected by the work environment or worker skill. In this paper, we discuss a detection method using image processing technology for detecting wire breakage, which is one of the most serious damages to wire ropes.

Keywords: Systems for Field Applications, Autonomous Vehicle Navigation, Machine Learning
Abstract: Gaussian Processes have been effectively used to learn location-to-signal-strength mappings from previously acquired observations and enable WiFi-based robot self-localization. However, the cubic computational cost for training and the quadratic cost for prediction with respect to the number of training points limits their scalability, particularly with large datasets necessary for outdoor environments. To reduce prediction cost we propose the use of Gaussian Gridmaps, a spatial representation that stores mean and variance predictions from Gaussian Processes into gridmaps. This approach reduces prediction computational cost to constant time, at the expense of some localization accuracy and increased memory usage. Our experiments demonstrate the feasibility of this method for outdoor localization and examine the impact of quantization and grid resolution on localization performance.