Keywords: Semiconductor Manufacturing, AI-Based Methods, Planning, Scheduling and Coordination
Abstract: AI-based Smart Manufacturing Systems (AISMS) incorporates various technologies, i.e., Internet of Things (IoT), big data analytics, system modeling, and Artificial Intelligence (AI). Such technologies are permeating different aspects of manufacturing industry and make it smart and capable of addressing challenges such as interoperability, decentralization, distributed control, real-time manufacturing process control, service orientation, and maintenance optimization. As one of the most sophisticated manufacturing industries, semiconductor industry has been actively adopting AISMS to boost productivities. This is a half-day workshop on semiconductor smart manufacturing technology workshop. The purpose of this workshop is to share with IEEE communities the recent advancement and development of semiconductor smart manufacturing technologies and relevant applications ranging from semiconductor tools scheduling, AI based defect detection and classification, smart equipment dispatch, intelligent process control, etc. The workshop aims to provide technical discussion forum for researchers from different fields and promote interdisciplinary and multidisciplinary research collaboration.

Keywords: Autonomous Agents, Planning, Scheduling and Coordination, Agent-Based Systems
Abstract: Multi-robot teams have been used in a wide range of applications, including surveillance, inspection, rescue, automation, and logistics. Due to the variety of critical components in these applications, the collaboration between agents in the robot team can quickly become a challenging problem, particularly when there is a variety of hardware, battery life, size, and functionalities of the robots that are moving in a dynamic environment. Because the robots are working in an dynamic environment, they need to dynamically change their behaviors to adapt to the state of the environment in a way that is fully coupled to the type of agent. For example, depending on the robot, some environmental constraints can be waived or become more restricted. The tasks need to be assigned and managed precisely to achieve the goals while minimizing the execution time and energy costs and avoiding collisions. Due to the collaboration among autonomous robots, robot team establishment introduce new requirements, new challenges, and new solutions to real-world problems. While many heterogeneous and autonomous robots are organized as a team to accomplish a mission, assigning a proper task to each robot, and evaluating their performance before acting is essential. Optimal task assignment can avoid failures and increase operating efficiency while the robots are executing their mission. Role-Based Collaboration (RBC) is a flexible strategy that can facilitate agent collaboration between agents in centralized or decentralized management by using the Environments – Classes, Agents, Roles, Groups, and Objects (E-CARGO) model. Research shows that the RBC methodology can be used to manage a robot team’s performance by optimizing task allocations. However, a critical part of RBC is the role assignment which requires a pertinent evaluation matrix, i.e., Q, that reflects the qualification of each agent for each role. This workshop will discuss related methodologies including RBC approaches and the E-CARGO model.

Keywords: Manipulation Planning
Abstract: This presentation focuses on our research work on developing tactile sensors and dry adhesion skins for robotic hands with dexterous and versatile capability for grasping and adaptive manipulation. It also presents an overview of exploratory solutions to modeling of hyper-elastic soft robots, distributed control of soft actuators (polymers or fluids), strategies for soft manipulation, and rapid prototyping and fabrication of the sensors and elastic robots. I will showcase the ability to adjust fingertip pose for better contact using sensor feedback, especially for top-side gripping onto a nearly flat surface (smooth or rough) of an object with firm attachment. I will show practical applications in industrial automation and discuss the recent developments throughout the robotics community advancing in this promising direction.

Keywords: Machine learning, AI-Based Methods, Big-Data and Data Mining
Abstract: Machine learning (ML) is changing the world, and in particular, the world of automation. So far, this wave of ML research has also influenced the main themes at IEEE CASE 2018-2021: Knowledge-based Automation, Smart Automation, Automation Analytics, and Data-Driven Automation. The critical question, however, is: How much groundbreaking ML research has been performed in our community in recent years? Are we leading actors, or more followers, applying what others have already formulated? An AdHoc on Machine Learning for Automation has recently been initiated by the CASE steering committee. The goal is that CASE, T-ASE, and relevant TCs shall become important players in the tough scientific race around ML that is going on right now. This panel discussion will take that goal as a starting point, and then reason about how we can build strong automation-related ML research, by identifying organizational and infrastructural support, but also niche areas where our research community should take the lead. The goal is simply to achieve results that count, both concerning fundamental methodology development and applications in strategic areas, which cause not small but big improvements within the limited resources we still have on our common planet.

Keywords: Renewable Energy Sources, Energy and Environment-aware Automation, Power and Energy Systems automation
Abstract: Recent Wave Energy Converters are equipped with multiple legs and generators with an intention to maximize energy generation. When deployed, WEC is subjected to complex and varying wave patterns, and the controller must efficiently maximize the energy capture. Traditional controllers have shown limitations to meet these challenges. This paper introduces a Multi-Agent Reinforcement Learning controller (MARL), which outperforms the traditionally used spring damper controller. Our initial studies show that the complex nature of problems makes it is hard for training to converge. Hence, we propose a novel “skip training” approach which enables the MARL training to overcome performance saturation and converge to more optimum controllers compared to default MARL training, boosting power generation. We also present another novel hybrid training initialization (STHTI) approach, where the individual agents of the MARL controllers can be initially trained against the baseline Spring Damper (SD) controller individually and then be trained one agent at a time or all together in future iterations to accelerate convergence. We achieved double-digit gains in energy efficiency over the baseline Spring Damper controller with the proposed MARL controllers using the Asynchronous Advantage Actor-Critic (A3C) algorithm.

Keywords: Computer Vision in Automation, Logistics, Deep Learning in Robotics and Automation
Abstract: Autonomous bin picking poses significant challenges to vision-driven robotic systems given the complexity of the problem, ranging from various sensor modalities, to highly entangled object layouts, to diverse item properties and gripper types. Existing methods often address the problem from one perspective. Diverse items and complex bin scenes require diverse picking strategies together with advanced reasoning. As such, to build robust and effective machine-learning algorithms for solving this complex task requires significant amounts of comprehensive and high quality data. Collecting such data in real world would be too expensive and time prohibitive and therefore intractable from a scalability perspective. To tackle this big, diverse data problem, we take inspiration from the recent rise in the concept of metaverses, and introduce MetaGraspNet, a large-scale photo-realistic bin picking dataset constructed via physics-based metaverse synthesis. The proposed dataset contains 217k RGBD images across 82 different article types, with full annotations for object detection, amodal perception, keypoint detection, manipulation order and ambidextrous grasp labels for a parallel-jaw and vacuum gripper. We also provide a real dataset consisting of over 2.3k fully annotated high-quality RGBD images, divided into 5 levels of difficulties and an unseen object set to evaluate different object and layout properties. Finally, we conduct extensive experiments showing that our proposed vacuum seal model and synthetic dataset achieves state-of-the-art performance and generalizes to real world use-cases.

Keywords: Planning, Scheduling and Coordination, Task Planning, Cyber-physical Production Systems and Industry 4.0
Abstract: Aiming at the low efficiency and poor timeliness of the configuration on the mixed-model robotic assembly line, this paper proposes a digital twin-based virtual reconfiguration method for mixed-model robotic assembly line. By building a digital twin model of the physical assembly workshop, the interaction and integration between the physical assembly workshop and the virtual assembly workshop is promoted, and the rapid verification of the robotic assembly line reconfiguration solution is fulfilled. This paper establishes a mathematical model to minimize the reconfiguration cost of the mixed-model robotic assembly line, utilizes an adaptive neighborhood search bee algorithm (ANSBA), and designs multiple feasible solution generation operators and random solution generation operators to improve the convergence speed of the algorithm while avoiding falling into local optimum. Finally, the feasibility and effectiveness of the algorithm are verified by a case study of coupling mixed-model robotic assembly line reconfiguration.

Keywords: Agent-Based Systems, Manufacturing, Maintenance and Supply Chains, Planning, Scheduling and Coordination
Abstract: Due to the COVID-19 pandemic, the global supply chain is disrupted at an unprecedented scale under uncertain and unknown trends of labor shortage, high material prices, and changing travel or trade regulations. To stay competitive, enterprises desire agile and dynamic response strategies to quickly react to disruptions and recover supply-chain functions. Although both centralized and multi-agent approaches have been studied, their implementation requires prior knowledge of disruptions and agent-rule-based reasoning. In this paper, we introduce a model-based multi-agent framework that enables agent coordination and dynamic agent decision-making to respond to supply chain disruptions in an agile and effective manner. Through a small-scale simulated case study, we showcase the feasibility of the proposed approach under several disruption scenarios that affect a supply chain network differently, and analyze performance trade-offs between the proposed distributed and centralized methods.

Keywords: Agricultural Automation, Calibration and Identification, Sustainability and Green Automation
Abstract: Polyculture farming has environmental advantages but requires substantially more pruning than monoculture farming. We present novel hardware and algorithms for automated pruning. Using an overhead camera to collect data from a physical scale garden testbed, the autonomous system utilizes a learned Plant Phenotyping convolutional neural network and a Bounding Disk Tracking algorithm to evaluate the individual plant distribution and estimate the state of the garden each day. From this garden state, AlphaGardenSim selects plants to autonomously prune. A trained neural network detects and targets specific prune points on the plant. Two custom-designed pruning tools, compatible with a FarmBot gantry system, are experimentally evaluated and execute autonomous cuts through controlled algorithms. We present results for four 60-day garden cycles. Results suggest the system can autonomously achieve 0.94 normalized plant diversity with pruning shears while maintaining an average canopy coverage of 0.84 by the end of the cycles.

Keywords: Logistics, Planning, Scheduling and Coordination, Deep Learning Methods
Abstract: This paper presents a fast optimization method for the picking order sequence of automated order picking systems in logistics warehouses. In this order sequencing problem (OSP), the fulfillment sequence of the given picking order set is determined to optimize the performance measures such as makespan and deadlock occurrence. Simulation is generally necessary to evaluate these measures for complex automated systems. However, their order sequence cannot be optimized quickly due to the long calculation time. It may make the system productivity and flexibility lower than expected because its picking schedules cannot be updated frequently. We, therefore, propose a fast optimization method to solve these simulation-based OSPs by taking a pretrained surrogate-assisted optimization approach. Firstly, we utilized a Bayesian recurrent neural network (BRNN) as a surrogate model to accurately learn the relationship between picking order sequence and performances. Secondly, we developed the surrogate-assisted optimization method based on simulated annealing (SA) and BRNN. Numerical experiments show that the surrogate model can evaluate about 10000 times faster than simulation. The proposed method also obtains an optimized solution 8.9 times faster than simulation-based optimization by original SA.

Keywords: Additive Manufacturing, AI-Based Methods, Deep Learning in Robotics and Automation
Abstract: Laser metal deposition (LMD) is an additive manufacturing method for metal parts by using focused thermal energy to fuse materials as they are deposited. During LMD, transient thermal signatures such as the in-situ thermal images of melt pool, contain rich information about process performance. Early prediction of such transient thermal signatures provides opportunities for process monitoring and defect prevention. While physics-based models of LMD have been conventionally used for thermal signature prediction, they have limitations and are computationally expensive for real-time prediction. A scalable, efficient data-science-based model is therefore needed. This paper develops a deep-learning-based surrogate model, called LMD-cGAN, to predict and emulate the transient thermal signatures in LMD. The model generates images for the thermal dynamics of melt pool conditionally on the deposition layer. It enables early prediction of future-layer thermal signatures for an in-process part based on its early-layer thermal signatures. To respect the physics in LMD, a physics-guided image selection (PGIS) mechanism is integrated with LMD-cGAN to calibrate the predictions against physical benchmarks of transient melt pool for the process. The effectiveness and efficiency of the proposed method are demonstrated in a case study on the LMD of Ti-4Al-6V thin-walled structures.

Keywords: Additive Manufacturing, Process Control
Abstract: Despite its capability of fabricating highly complicated geometries, additive manufacturing (AM) or three-dimensional (3D) printing faces significant challenges in quality control, particularly the geometric accuracy of printed thin walls. Important to automotive, aerospace, and medical applications, this category of products tend to distort and deviate from their nominal designs. To establish a systematic accuracy modeling and control approach for 3D printed thin-wall structures, this study develops a small-sample learning approach using printing primitives. By treating each product as a combination of printing primitives, we overcome the small-data challenge by transforming a small set of training products into a large sample of geometric primitives with covariates of sizes and locations. To incorporate the process knowledge, we model the stack-up of primitives into thin walls through a convolution formulation of AM processes. A real case study shows the promise of the proposed small-sample learning method for accuracy control of 3D printed thin walls.

Keywords: Additive Manufacturing, Planning, Scheduling and Coordination
Abstract: Material extrusion additive manufacturing is an essential technology for rapid prototyping. The standard approach to planning the deposition toolpath for this technology builds each layer sequentially. Unfortunately this approach typically results in significant wasted motion, which is a barrier for use in industrial production. We give a new method for toolpath planning that improves on the layer-based approach as well as our own previous methods that build toolpaths across layers. In this paper we present a new method of toolpath planning that utilizes a Reeb decomposition on the input model. This geometric decomposition allows toolpath planning over subcomponents of the model, rather than over individual extrusion segments. This allows a top-down construction of toolpaths, and is highly effective. We test our new approach, which we call Reeb planning, over a benchmark of 50 models and achieve a reduction of 49.7% in wasted motion over standard layer-based methods. Our decomposition scheme also provides insight into model classification, which can be used for improved production planning.

Keywords: Additive Manufacturing, Probability and Statistical Methods, Machine learning
Abstract: In-situ measurements provide vast information for additive manufacturing process understanding and real-time control. Data from various monitoring techniques observes different characteristics of a build process. Fusing multi-modal in-situ monitoring data can significantly enhance process anomaly detection, part defect prediction and build failure diagnosis, thus improve AM part quality control. This paper compares the powder bed fusion in-process observations from two types of AM in-situ monitoring, coaxial melt pool imaging and layerwise imaging, and investigates the correlation between the two observations for a build of parts with multiple geometric features and scan patterns. All data were collected from an open architecture powder bed fusion AM testbed. Data analysis shows that both datasets exhibit significant statistical changes when new features introduced during the build. However, further machine learning based modeling indicates that statistical features extracted from the two data sets do not correlate very well. Discussions are provided on how the statistical analysis of the observations from the two modality monitoring system can be utilized for data fusion strategy development, especially toward improving process anomaly detection.

Keywords: Probability and Statistical Methods, Product Design, Development and Prototyping, AI-Based Methods
Abstract: Additive manufacturing (AM) provides a higher level of flexibility to build customized products with complex geometries, by selectively melting and solidifying metal powders. However, wide applications of AM beyond rapid prototyping are currently limited by its ability to perform quality assurance and control. Advanced melt-pool monitoring provides a unique opportunity to increase information visibility in the AM process. Stochastic melt-pool variations are closely pertinent to the quality of an AM build. There is a pressing need to investigate the variances of melt pools along the temporal scanning path, as well as within a 3D spatial neighborhood of the focal point by the laser beam. This paper presents a stochastic modeling framework for statistical modeling and monitoring of melt-pool imaging data, including tensor decomposition of high-dimensional data, additive Gaussian process modeling of low-dimensional profiles as random variables, and hypothesis testing via the construction of confidence boundary. Experimental results show the effectiveness of tensor decomposition for spatiotemporal monitoring of melt-pool variations in the metal-based AM process.

Keywords: Industrial Robots, Control Architectures and Programming, Manufacturing, Maintenance and Supply Chains
Abstract: This paper considers the trajectory tracking con- trol for wire arc additive manufacturing (WAAM). Some aspects of the WAAM process, such as the angles between welding torch direction, deposition surface normal and gravity are very important to ensure the quality of the manufactured parts. A task-priority based kinematic control scheme is proposed to control a manipulator and a positioning table coordinately. The manipulator and the positioning table are considered as a single robotic chain, and the primary task is defined to make the welding torch follow a desired trajectory defined in the positioning table deposition frame, while the secondary task is to align the welding torch in a desired direction defined in the inertial frame, e.g., gravity direction. A complete Lyapunov stability analysis is performed considering unmodeled dynamics in the kinematic control loop. The effectiveness of the proposed method is shown experimentally on a WAAM robotic system composed of a 6-axis industrial manipulator, a 2-axis positioning table and a Cold Metal Transfer (CMT) power source.

Keywords: Cyber-physical Production Systems and Industry 4.0, Agent-Based Systems, Factory Automation
Abstract: In recent years, economic changes driven by an increase in consumer and small business demand for highly customised and small-batch products have necessitated a paradigm shift away from traditionally rigid manufacturing towards a more agile and decentralised approach to factory control. Such an approach, by eliminating centralised control, gives rise to factories whose production capabilities can be reconfigured at a moment’s notice in order to respond to high-frequency shifts in production demand. This paper proposes a novel architecture for decentralised factory control based on multi-agent technologies. Unlike the previous research work in this field, this research differs in its being designed from the ground up using open-source software, instead of relying on existing solutions for agent-to-agent communications. The presented architecture allows for real-time production flexibility in response to changes in the factory layout, e.g. machines going offline, or new machines being added. It achieves this with a ping-response system. A piece of software has been developed to simulate the functions of the proposed architecture.

Keywords: Cyber-physical Production Systems and Industry 4.0, Factory Automation, Intelligent and Flexible Manufacturing
Abstract: In recent years, various abstract and practical architectures have been proposed in the context of Industry 4.0. While these architectures focus on different aspects, their common goal is to facilitate the transformation of a static production into a flexible, resilient, interconnected cyber-physical production system (CPPS). However, reviewing those reveals that a general procedure for applying those architectures is missing. In this paper, we propose a modular, scalable and lightweight architecture utilizing simplified semantic information models. We also present an integration guide that helps factory owners and process engineers to apply the proposed architecture. Furthermore, we also show how the factory operators can make architectural decisions according to their needs. This approach will help speed up the application of the architecture for the realization of a modular and scalable CPPS.

Keywords: Cyber-physical Production Systems and Industry 4.0, Intelligent and Flexible Manufacturing
Abstract: The development of modern automated production systems requires the close cooperation of engineers from different domains. Due to the large amount of domain-specific documents and heterogeneous data they create during the multidisciplinary engineering activities, ensuring the consistency of information is always challenging. Since most of these documents are texted-based and lack a standardized structure, extracting required information from these files is oftentimes problematic. This issue is particularly critical in the development of large-scale production plants due to the high complexity of the systems and the diversity of disciplines involved. To help engineers efficiently utilize unstructured data sources as well as identify potential information contradictions, we propose an ontology-based inconsistency management approach for large-scale production systems that generates the knowledge base from unstructured engineering data and (semi-) automatically detects multiple types of inconsistencies. In addition, the presented framework also supports the tracking of information changes during the system design process.

Keywords: Cyber-physical Production Systems and Industry 4.0, Intelligent and Flexible Manufacturing
Abstract: Many core technologies of Industry 4.0 have gained substantial advancement in recent years. Digital Twin (DT) has become the key technology and tool for manufacturing industries to realize intelligent cyber-physical integration and digital transformation by leveraging these technologies. Although there have been many DT-related works, there is no standard definition, unified framework, and implementation approach to DT until now. Widely developing DTs for the manufacturing industry is still challenging. Thus, this paper proposes a novel implementation framework of digital twins for intelligent manufacturing, denoted as IF-DTiM, which possesses several distinct merits to distinguish itself from previous works. First, IF-DTiM fully utilizes new-generation container technology so that DT-related applications and services can be packaged in a self-contained way, rapidly deployed, and robustly operated with the capabilities of failover, autoscaling, and load balancing. Second, it leverages existing intelligent cloud manufacturing services to realize the intelligence for DT externally in a scalable and plug-and-play manner instead of using traditional approaches to embed intelligence in DT. Third, IF-DTiM contains Product DT for products, Equipment DT (i.e., EQ DT) for equipment, and Process DT for production lines, which can generically fulfill the demands and scenarios to achieve intelligent manufacturing for various manufacturing industries. Testing results show that IF-DTiM can achieve remarkable performance in rapid deployment and real-time data exchanges of DT-related applications. Finally, we develop an example DTiM system for CNC machining based on IF-DTiM to demonstrate its efficacy and applicability in facilitating the manufacturing industry to build their DT system

Keywords: Cyber-physical Production Systems and Industry 4.0, Manufacturing, Maintenance and Supply Chains, Factory Automation
Abstract: With an increase in demand for individualized and personalized products, manufacturers are turning their attention to more flexible manufacturing systems that can be rapidly reconfigured, based on needs. However, the existing approaches primarily rely on manual reconfiguration performed or managed by subject-matter experts, which is time-consuming and labor-intensive. To this end, we propose an integrated framework that generates multiple feasible configurations, conducts simulations to evaluate the performance of the proposed configurations, and performs a multi-objective optimization to derive a set of ordered solutions from which the manufacturer may select their desired option. The framework consists of three core components: textit{Digital Twin Pool}, textit{Application Plane}, and textit{Decision Maker}. The DT pool consists of DTs grouped together based on functionalities. The individual DT request required information from different applications in the Application Plane. The applications include a semantic-based ontology map for knowledge representation and storage, and a simulation application for simulating generated line configurations to obtain necessary attribute values such as throughput, yield, cycle times, etc. The Decision Maker includes an optimizer, which takes multiple configurations obtained from the DT pool and runs a multi-objective optimization. The output of the Decision Maker is a set of feasible solutions that will be provided to the user. A case study is presented to demonstrate the efficacy and usefulness of the proposed framework.

Keywords: Cyber-physical Production Systems and Industry 4.0, Robust/Adaptive Control, Failure Detection and Recovery
Abstract: The Overall Equipment Effectiveness (OEE) of automated Production Systems (aPS) is an essential metric in a competitive global market. Maintaining high OEE is not easily achieved in aPS, because hard real-time requirements and the fulfillment of extra-functional tasks, including safety, limit the flexibility needed for fault-tolerant control and dynamic reconfiguration. This paper proposes the extension of aPS by a distributive agent to monitor the process and intervene only in case of potential OEE loss, leaving the core system unaltered. The approach's viability is demonstrated using a preexisting demonstrator machine whose OEE is monitored by a physically separate agent, connected via the internet. Unbound by real-time requirements, this agent monitors potential risks to OEE and, if deemed necessary, automatically intervenes with instructions to the operative layer to trigger a reconfiguration. This work was submitted alongside another paper, conceptualizing the inclusion of component health modeling and observation in the distributive agent's capabilities.

Keywords: Calibration and Identification, Assembly, Factory Automation
Abstract: To enable robots to grasp unknown objects, we have developed a new type of fingertip-mounted sensor that can detect distance, material type, and interior structure without making contact with the object to be grasped. Due to its working principle, the sensor is named as Dual-Modal and Dual Sensing Mechanisms (DMDSM) pretouch sensor. To enable the wide deployment of the DMDSM sensor, we need to scan a large number of common household items using the sensor to establish an object/material database. Here we report our progress in designing an automatic object scanning system and the sensor calibration algorithm with the new sensor. The object scanning system is constructed by a refitted 3D printer with a motorized turntable mounted on its printing stage. The extruder of the 3D printer is replaced by the sensor to perform 3D translation. The turntable rotates the object of interest to allow a full-body scan. A prototype of the scanning system has been built, and a new calibration algorithm has been developed to estimate the parameters of both the sensor and the scanning system. The system design and ranging accuracy have been verified by physical experiments, and the collected data from seven types of common household objects have shown promising prospects of using DMDSM sensors in grasping.

Keywords: Calibration and Identification, Cyber-physical Production Systems and Industry 4.0, Industrial and Service Robotics
Abstract: Accurately solving the problem of hand-eye calibration is a fundamental requirement for high-precision robotic applications involving a vision sensor. Particularly in the case of 2D laser profile scanners, its complexity increases since individual 2D measurements are not sufficient to calculate the positional and rotational relationship between tool and scanner. In this work, we propose a novel method to solve the hand-eye calibration problem for a robot-guided 2D laser sensor. Our approach is easy to implement, does not require expert knowledge, uses everyday objects instead of expensive calibration pieces, and runs automatically with minimal user interaction. Furthermore, we present an algorithm to automatically determine an optimized set of view poses. We evaluate our approach on a welding robot application and compare the results to existing methods. Our method achieves an accuracy of 0.3 mm in terms of a mean absolute error metric in the robot cell shown in Fig. 1.

Keywords: Calibration and Identification, Model Learning for Control, Task Planning
Abstract: An object's inertial parameters, such as the mass, the center of mass (CoM), and the moment of inertia, affect the response to the external forces exerted on it. It is important to estimate these parameters in order to facilitate robot-led automation including grasping and manipulation. Traditionally, the estimation is conducted by a specific hardware in a controlled environment, which may not be always available for a robotic system. We propose an efficient novel method for estimating the CoM of an object via force sensor-less dynamic pushing and a vision sensor detecting the change in the object's pose. The simulation results showed that the proposed method achieved an accurate and stable estimation under both the unknown isotropic and anisotropic floor friction conditions.

Keywords: Embedded Systems for Robotic and Automation, Sensor Networks
Abstract: This study develops a multi-hypothesis extended Kalman filter (MH-EKF) for the online estimation of the bending angle of a 3D printed soft sensor attached to soft actuators. Despite the advantage of compliance and low interference, the 3D printed soft sensor is susceptible to the hysteresis property and nonlinear effects. Improving measurement accuracy for sensors with hysteresis is a common challenge. Current studies mainly apply complex models and highly nonlinear functions to characterize the hysteresis, requiring a complicated parameter identification process and challenging real-time applications. This study enhances the model simplicity and the real-time performance for the hysteresis characterization. We identify the hysteresis by combining multiple polynomial functions and improving the sensor estimation with the proposed MH-EKF. We examine the performance of the filter in the real-time closed-loop control system. Compared with the baseline methods, the proposed approach shows improvements in the estimation accuracy with low computational complexity.

Keywords: AI-Based Methods, Computer Vision for Automation, Software Architecture for Robotic and Automation
Abstract: In the proposed study, we describe an approach to improving the computational efficiency and robustness of visual SLAM algorithms on mobile robots with multiple cameras and limited computational power by implementing an intermediate layer between the cameras and the SLAM pipeline. In this layer, the images are classified using a ResNet18-based neural network regarding their applicability to the robot localization. The network is trained on a six-camera dataset collected in the campus of the Skolkovo Institute of Science and Technology (Skoltech). For training, we use the images and ORB features that were successfully matched with subsequent frame of the same camera (“good” keypoints or features). The results have shown that the network is able to accurately determine the optimal images for ORB-SLAM2, and implementing the proposed approach in the SLAM pipeline can help significantly increase the number of images the SLAM algorithm can localize on, and improve the overall robustness of visual SLAM. The experiments on operation time state that the proposed approach is at least 6 times faster compared to using ORB extractor and feature matcher when operated on CPU, and more than 30 times faster when run on GPU. The network evaluation has shown at least 90% accuracy in recognizing images with a big number of “good” ORB keypoints. The use of the proposed approach allowed to maintain a high number of features throughout the dataset by robustly switching from cameras with feature-poor streams.

Keywords: Cognitive Automation, Learning and Adaptive Systems, Computer Vision in Automation
Abstract: This paper describes a method for generating robot grasps by jointly considering stability and other task and object-specific constraints. We introduce a three-level representation that is acquired for each object class from a small number of exemplars of objects, tasks, and relevant grasps. The representation encodes task-specific knowledge for each object class as a relationship between a keypoint skeleton and suitable grasp points that is preserved despite intra-class variations in scale and orientation. The learned models are queried at run time by a simple sampling-based method to guide the generation of grasps that balance task and stability constraints. We ground and evaluate our method in the context of a Franka Emika Panda robot assisting a human in picking tabletop objects for which the robot does not have prior CAD models. Experimental results demonstrate that in comparison with a baseline method that only focuses on stability, our method jointly considers additional constraints to provide suitable grasps for different tasks.

Keywords: Computer Vision for Manufacturing, Computer Vision in Automation
Abstract: The insertion of through-hole components is a difficult task. As the tolerances of the holes are very small, minor errors in the insertion will result in failures. These failures can damage components and will require manual intervention for recovery. Errors can occur both from imprecise object grasps and bent pins. Therefore, it is important that a system can accurately determine the object's position and reject components with bent pins. By utilizing the constraints inherent in the object grasp a method using template matching is able to obtain very precise pose estimates. Methods for pin-checking are also implemented, compared, and a successful method is shown. The set-up is performed automatically, with two novel contributions. A deep learning segmentation of the pins is performed and the inspection pose is found by simulation. From the inspection pose and the segmented pins, the templates for pose estimation and pin check are then generated. To train the deep learning method a dataset of segmented through-hole components is created. The network shows a 97.3 % accuracy on the test set. The pin-segmentation network is also tested on the insertion CAD models and successfully segment the pins. The complete system is tested on three different objects, and experiments show that the system is able to insert all objects successfully. Both by correcting in-hand grasp errors and rejecting objects with bent pins.

Keywords: Computer Vision for Manufacturing, Deep Learning Methods, AI-Based Methods
Abstract: Learning-based grasping models typically require a large amount of training data and training time to generate an effective grasping model. Alternatively, small non-generic grasp models have been proposed that are tailored to specific objects by, for example, directly predicting the object's location in 2/3D space, and determining suitable grasp poses by post processing. In both cases, data generation is a bottleneck, as this needs to be separately collected and annotated for each individual object and image. In this work, we tackle these issues and propose a grasping model that is developed in four main steps: 1. Visual object grasp demonstration, 2. Data augmentation, 3. Grasp detection model training and 4. Robot grasping action. Four different vision-based grasp models are evaluated with industrial and 3D printed objects, robot and standard gripper, in both simulation and real environments. The grasping model is implemented in the OpenDR toolkit at: https://github.com/opendr-eu/opendr/tree/master/projects/control/single _demo_grasp.

Keywords: Computer Vision for Manufacturing, Zero-Defect Manufacturing, Factory Automation
Abstract: Here, the thickness distribution of a spray-painted film was analyzed by computational fluid dynamics, considering the change in the paint shape due to flow. We focused on the paint adhering to the target because this behavior has not been previously examined. The particle method was adopted for the calculation because it enabled a stable analysis of the paint droplets and the complex uneven surface of the coating film. A high-speed camera and image analysis were used to capture the spray painting and identify the values of the parameters. Using the developed model, we analyzed the change in the film thickness distribution for the scene of painting on a flat plate in the vertical direction. It was confirmed that the numerical and experimental data correlated for two conditions of the target distance.

Keywords: Computer Vision in Automation, Computer Vision for Manufacturing
Abstract: Many industrial assembly tasks involve peg-in-hole like insertions with sub-millimeter tolerances which are challenging, even in highly calibrated robot cells. Visual servoing can be employed to increase the robustness towards uncertainties in the system, however, state of the art methods either rely on accurate 3D models for synthetic renderings or manual involvement in acquisition of training data. We present a novel self-supervised visual servoing method for high precision peg-in-hole insertion, which is fully automated and does not rely on synthetic data. We demonstrate its applicability for insertion of electronic components into a printed circuit board with tight tolerances. We show that peg-in-hole insertion can be drastically sped up by preceding a robust but slow force-based insertion strategy with our proposed visual servoing method, the configuration of which is fully autonomous.

Keywords: Sensor Fusion, Deep Learning in Robotics and Automation, Computer Vision for Transportation
Abstract: Image and Point Clouds provide different information for robots. Finding the correspondences between data from different sensors is crucial for various tasks such as localization, mapping, and navigation. Learning-based descriptors have been developed for single sensors; there is little work on cross-modal features. This work treats learning cross-modal features as a dense contrastive learning problem. We propose a Tuple-Circle loss function for cross-modality feature learning. Furthermore, to learn good features and not lose generality, we developed a variant of widely used PointNet++ architecture for point cloud and U-Net CNN architecture for images. Moreover, we conduct experiments on a real-world dataset to show the effectiveness of our loss function and network structure. We show that our models indeed learn information from both images as well as LiDAR by visualizing the features.

Keywords: Intelligent and Flexible Manufacturing, Deep Learning Methods, Computer Vision for Manufacturing
Abstract: The problem concerned is accurate monitoring of the penetration, in a fully penetrated weld pool, as quantified by the width of the weld on the back-side of the workpiece. A popular method is to use a weld pool image to derive it. Analysis of the physical process suggests that a single weld pool does not contain adequate information but most recent serial weld pools may. To this end, a CNN-LSTM (convolutional neural network combined with long-short term memory one) model is proposed. Dynamic weld pools are experimentally generated using changing the welding current and welding speed randomly. The weld pools are imaged using an HDR camera during experiments. Images are also captured from the back-side surface of the workpiece to provide the ground truth for training, validation, and testing. It is found that the highly dynamically changing weld pool can be accurately predicted using serial weld pool images at 0.3 mm for its back-side bead width. Comparison has been made with results from comparative studies to verify the effectiveness of and the contribution from the information adequacy (by using serial images) and the feature extracting capability (by using deep learning).

Keywords: Manipulation Planning
Abstract: Robotic manipulation of micro-objects is usually disturbed by forces that are predominant at the micro-scale such as capillary and electrostatic forces. Dexterous micro-manipulation methods that consider these effects are currently limited to planar manipulation of complex objects or 3-D manipulation of objects with simple shapes. This paper presents an original method able to generate optimal fingers trajectories for 3-D in-hand micro-manipulation of complex micro-objects. Since adhesion forces in micro-scale are usually difficult to predict, the proposed method considers the worst case where adhesion forces are overestimated when they have a disturbing effect and neglected when they contribute to stabilising the grasps. A graph of interconnected stable and accessible grasps is then built. Finally, the optimal trajectories are generated using the A* algorithm. When compared to the case where adhesion forces are neglected, the simulation results show that the computation time is shorter while the generated trajectories require more operations because of the excluded risky grasps.

Keywords: Agent-Based Systems, Planning, Scheduling and Coordination
Abstract: In this letter, we develop a distributed consensus-based online monitoring framework for a robot swarm with a fixed graph structure. Each agent can monitor whether the swarm satisfies specifications given in the form of Swarm Signal Temporal Logic (SwarmSTL) formulas. SwarmSTL formulas describe temporal properties of swarm-level features represented by generalized moments (GMs), e.g., centroid and variance. To deal with measurement noise, we propose a generalized moment consensus algorithm (GMCA) with Kalman filter (KF), allowing each agent to estimate the GMs. Besides, we prove the convergence properties of the GMCA and derive an upper bound for the error between an agent’s estimate of the GMs and the actual GMs. This upper bound is derived to be dependent on the maximal allowed velocity but independent of the agents’ exact motion. A set of distributed monitoring rules for SwarmSTL formulas are proposed based on the estimation error bound. As a result, the agents can monitor the satisfaction of SwarmSTL formulas over swarm features during execution. The distributed monitoring framework is applied to a supply transportation example, where the efficacy of KF in the GMCA is also shown.

Keywords: Agent-Based Systems, Planning, Scheduling and Coordination, Factory Automation
Abstract: Industrial environments that rely on Mass Customization are characterized by high variety of product models and reduced batch sizes, demanding prompt adaptation of resources to a new product model. In such environment, it is important to schedule tasks that require manual procedures with different levels of complexity and repetitiveness. In a garment mass customization scenario, task scheduling needs to take into consideration the dependency of the tasks, meaning that in order to initiate a certain task, materials from previous tasks may be required. In order to carry out a smooth scheduling process within a garment mass customization factory, not only the tasks but also the transportation of materials to perform such tasks need to be scheduled to static and mobile robots, respectively. To tackle this problem, we propose a set of heuristics that are able to schedule both the task work and transportation of materials. We analyze these heuristics theoretically with respect to computational complexity. Subsequently, the performance of each algorithm is evaluated using a synthetic testset. The comparative analysis shows that the extended algorithms have close results among themselves, whereas for the heuristics, MTC outperforms all of the other algorithms. Moreover, the combination of PEFT and MTC is more efficient compared to other algorithm combinations.

Keywords: Probability and Statistical Methods, Intelligent Transportation Systems
Abstract: Precise tracking of haul truck activities in mining aids automation and data analysis and allows for accurate material tracking. Fleet Management Systems (FMS) record information about haul truck activities, however, today's mines frequently employ legacy systems with limited sensing leading to poor data quality. In this letter we present a novel method for load-haul cycle segmentation using vehicle telemetry data to determine precise load and dump times. Central to this is a Hidden Semi-Markov Model (HSMM) which infers a truck's state from discrete observations generated from GPS positional data. The HSMM is trained unsupervised from historic data. Evaluation of segmented state estimates made from real-world data showed over 98% of loads and 91% of dumps were correctly identified, demonstrating the effectiveness of the proposed method.

Keywords: Energy and Environment-aware Automation, Optimization and Optimal Control, Power and Energy Systems automation
Abstract: The fuel economy of a hybrid electric vehicle (HEV) is determined by its energy management strategy (EMS), while the conventional EMS usually suffers from enormous computation loads when solving a nonlinear optimization problem. To resolve this issue, this paper presents a computationally efficient EMS with close-to-optimal performance using very limited computation resources. Relying on the optimal solutions by offline dynamic programming (DP), a constrained model predictive control (MPC) can quickly determine the engine on/off status and then the torque split problem is solved by a value-based Pontryagin’s minimum principle (PMP). Two measures are taken to further reduce the online computation cost: by surface fitting, the tabular value function is replaced by piecewise linear polynomials and thus the memory occupation is greatly reduced; and by model approximation, the nonlinear torque split problem becomes a quadratic programming one that can be more rapidly solved. The testing results from processor-in-the-loop (PIL) simulation indicate that the proposed EMS can generate a fuel efficiency close to the one by DP, but saves 70% onboard memory space and 30% CPU utilization compared with the benchmark EMS without taking the two measures.

Keywords: Agricultural Automation, Robotics and Automation in Agriculture and Forestry
Abstract: Weed competition is one of the most limiting factors affecting crop yield and profitability. Robotic weeding systems have demonstrated their potential to save herbicide usage and thereby minimize costs and adverse impacts on the environment. We introduce the software and hardware design of an automatic system for micro-dose herbicide spray using a mobile robot for early-stage weed control. The system is equipped with a stereo camera, one inertial measurement unit, and multiple linearly actuating spray nozzles. To enable the system, we propose a new scene representation from the perspective of spray operation. We represent the space occupied by weeds as candidate line segments for spray and then construct a directed acyclic graph (DAG) that embraces the feasible nozzle paths among weeds. Based on the new scene representation, we formulate an optimal K-nozzle assignment/motion planning problem and develop a binary linear programming-based algorithm to assign nozzles to the candidate line segments for optimal coverage. We built the system and conducted both simulation and field experiments. Evaluation on rough soil surface with artificial targets has shown that the lateral errors of herbicide spray are at sub-centimeter levels. Simulation results demonstrate that the proposed assignment algorithm can provide good coverage within the intra-row regions

Keywords: Agricultural Automation
Abstract: Autonomous crop monitoring is a difficult task due to the complex structure of plants. Occlusions from leaves can make it impossible to obtain complete views about all fruits of, e.g., pepper plants. Therefore, accurately estimating the shape and volume of fruits from partial information is crucial to enable further advanced automation tasks such as yield estimation and automated fruit picking. In this paper, we present an approach for mapping fruits on plants and estimating their shape by matching superellipsoids. Our system segments fruits in images and uses their masks to generate point clouds of the fruits. To combine sequences of acquired point clouds, we utilize a real-time 3D mapping framework and build up a fruit map based on truncated signed distance fields. We cluster fruits from this map and use optimized superellipsoids for matching to obtain accurate shape estimates. In our experiments, we show in various simulated scenarios with a robotic arm equipped with an RGB-D camera that our approach can accurately estimate fruit volumes. Additionally, we provide qualitative results of estimated fruit shapes from data recorded in a commercial glasshouse environment.

Keywords: Agricultural Automation, Autonomous Vehicle Navigation, Reinforcement
Abstract: Precision agriculture is rapidly attracting research to efficiently introduce automation and robotics solutions to support agricultural activities. Robotic navigation in vineyards and orchards offers competitive advantages in autonomously monitoring and easily accessing crops for harvesting, spraying and performing time-consuming necessary tasks. Nowadays, autonomous navigation algorithms exploit expensive sensors which also require heavy computational cost for data processing. Nonetheless, vineyard rows represent a challenging outdoor scenario where GPS and Visual Odometry techniques often struggle to provide reliable positioning information. In this work, we combine Edge AI with Deep Reinforcement Learning to propose a cutting-edge lightweight solution to tackle the problem of autonomous vineyard navigation without exploiting precise localization data and overcoming task-tailored algorithms with a flexible learning-based approach. We train an end-to-end sensorimotor agent which directly maps noisy depth images and position-agnostic robot state information to velocity commands and guides the robot to the end of a row, continuously adjusting its heading for a collision-free central trajectory. Our extensive experimentation in realistic simulated vineyards demonstrates the effectiveness of our solution and the generalization capabilities of our agent.

Keywords: Optimization and Optimal Control, Energy and Environment-aware Automation, Automation Technologies for Smart Cities
Abstract: This paper is devoted to the development of an optimal speed profile for autonomous vehicles in order to cross a signalized intersection without stopping. The design objective is to achieve both a short travel time and low energy consumption by taking full advantage of the traffic light information based on vehicle-to-infrastructure communication. The eco-driving problem is formulated as an optimal control problem. For the case where the vehicles are in free-flow mode, we derive a real-time on-line analytical solution, distinguishing our method from most existing approaches based on numerical calculations. Under mild assumptions, the optimal eco-driving algorithm is readily extended to cases where the free-flow mode does not apply due to the presence of interfering traffic. Extensive simulations are provided to compare the performance of autonomous vehicles under the proposed speed profile and human-driven vehicles. The results show quantitatively the advantages of the proposed algorithm in terms of energy consumption and travel time.

Keywords: Agricultural Automation, Industrial and Service Robotics
Abstract: In this paper, we present a novel prototype for produce harvesting, that fulfils a set of predefined objectives, namely being productive, versatile, and robust. In our approach, the produce of interest enters the harvesting tube at one end, and exits from the other, in a way to cut the time consumed in manipulation. The stem is being cut by a laser module, with the optics set up for a faraway focal point. Such arrangement allows for minimal hardware at the produce-entry side and in turn, the overall dimensions. This is essential for fruit reachability. The tube is fitted to a robot manipulator to approach the harvest and maneuver for stem positioning into the focal point of the laser beam. An in-tube vision system is designed for stem/sepal and produce detection with utmost robustness against ambient light conditions. Successful experiments have been conducted by sample harvesting strawberry, chilli pepper, and aubergine as evident of versatility towards produce geometry and texture.

Keywords: Agricultural Automation, Motion and Path Planning, Collision Avoidance
Abstract: The majority of motion planning strategies developed over the literature for reaching an object in clutter are applied to two dimensional (2-d) space where the state space of the environment is constrained in one direction. Fewer works have been investigated to reach a target in 3-d cluttered space, and when so, they have limited performance when applied to complex cases. In this work, we propose a constrained multiobjective optimization framework (OptI-ProMP) to approach the problem of reaching a target in a compact clutter with a case study on soft fruits grown in clusters, leveraging the local optimisation-based planner CHOMP. OptI-ProMP features costs related to both static, dynamic and pushable objects in the target neighborhood, and it relies on probabilistic primitives for problem initialisation. We tested, in a simulated poly-tunnel, both ProMP-based planners from literature and the OptI-ProMP, on low (3-dofs) and high (7-dofs) dexterity robot body, respectively. Results show collision and pushing costs minimisation with 7-dofs robot kinematics, in addition to successful static obstacles avoidance and systematic drifting from the pushable objects center of mass.

Keywords: Optimization and Optimal Control, Autonomous Agents, Planning, Scheduling and Coordination
Abstract: We focus on the problem of optimally managing a set of unmanned aerial vehicles performing given missions that require to land on an automatic platform, unmount the currently-carried payload, and take off with another payload to complete mission objectives. Such a problem often arises when swarms of drones cooperate to complete monitoring applications or other tasks requiring an efficient schedule of landings and payload switches in a resource-constrained environment. First, the problem is formulated as a mixed-integer linear programming one, which, however, may be complex to be solved for a large number of drones. Thus, we also propose a heuristic algorithm able to find suboptimal solutions with a reduced computational effort. Preliminary simulation results are reported and discussed.

Keywords: Automation at Micro-Nano Scales, AI-Based Methods, Nanomanufacturing
Abstract: The laws of physics are usually stated using mathematical equations, allowing us to accurately map a given physical system to its response. However, when building systems, we are often faced with the inverse problem: How should we design a physical system that produces a target response? In this paper, we present a novel algorithm, called FastPivot, for solving such inverse problems. FastPivot starts with a system state and invokes alternating forward and backward passes through the system variables. In a forward pass, it leads the current state of the system to its response. In the subsequent backward pass, a small amount of information is allowed to percolate from the target response back to the system variables. FastPivot produces good quality solutions efficiently. We demonstrate the promise of FastPivot on the inverse problem of placing atoms in a bounded region using a scanning tunneling microscope to achieve target responses in the density of states. We also compare FastPivot to Monte Carlo methods and analyze various empirical observations.

Keywords: Automation at Micro-Nano Scales, Manipulation Planning, Motion and Path Planning
Abstract: Online manipulation of multiple micro- and nanoscale agents is of major interest for various research applications. Among the biggest limitations of wireless external actuation are its global and coupled influences in the workspace, which limit the robust manipulation of multiple agents independently and simultaneously. In this paper, we propose novel motion planning algorithms, Bi-iSST and Ref-iSST, to quickly generate time-optimal trajectories for multiple agents sharing global external fields. Both algorithms are extended by the stable sparse rapidly-exploring random tree kinodynamic motion planning algorithm. The Bi-iSST uses a bidirectional approach to speed up the searching process. A novel connection process is proposed to connect the two trees efficiently by applying an optimization procedure. The Ref-iSST uses the workspace information to quickly generate global-routing trajectories as references, then guides the search process more effectively by getting more accurate heuristics according to the reference global-routing trajectories. A transition matrix similar to that in Markov Decision Processes is used to form the reference trajectory. Compared with the state-of-the-art iSST algorithm, the proposed algorithms quickly update feasible solutions and converge to a near-optimal, minimum-time solution to increase the efficiency of the simultaneous manipulation of multiple micro agents using global external fields. Extensive analysis and physical experiments are presented to confirm the effectiveness and the performance of the motion planning algorithms.

Keywords: Embedded Systems in Meso, Micro and Nano Scale, Automation at Micro-Nano Scales, Collision Avoidance
Abstract: An important problem in microrobotics is how to control a large group of microrobots with a global control signal. This paper focuses on controlling a large-scale swarm of MicroStressBots with on-board physical finite-state machines. We introduce the concept of group-based control, which makes it possible to scale up the swarm size while reducing the complexity both of robot fabrication as well as swarm control. We prove that the group-based control system is locally accessible in terms of the robot positions. We further hypothesize based on extensive simulations that the system is globally controllable. A nonlinear optimization strategy is proposed to control the swarm by minimizing control effort. We also propose a probabilistically complete collision avoidance method that is suitable for online use. The paper concludes with an evaluation of the proposed methods in simulations.

Keywords: Industrial Robots, Automation at Micro-Nano Scales
Abstract: The current study presents an adaptive control approach to solve the tracking trajectory problem for a robotic manipulator that uses a gripper based on bimorph piezolectric actuators. The development of an adaptive gain state feedback form, that considers the state restrictions, is proposed using a novel class of barrier Lyapunov function that drive the effective control of joints and piezolectric actuators. The proposed method allows to include complex combinations of state restrictions in the Lyapunov function yielding the construction of differential forms for the gains in the controller that can handle the evolution of trajectories of the robotic arm inside the restricted region. The proposed control design success tracking reference trajectories for both the joints of the robotic arm, as well as the motion of the piezolectric device, during several operative scenarios. A comprehensive numerical study evaluates the effect of introducing the state dependent gain considering the state restrictions of elliposidal type. The comparison of the mean square error confirms the contributions of the developed control action, showing better tracking quality, and a smaller control power along the same evaluation. This comparison seems the validate the contribution of the proposed controller with respect to a feedback form with fixed gains.

Keywords: Planning, Scheduling and Coordination
Abstract: This paper addresses a type of rendezvous recharging problem involving a team of unmanned aerial and ground vehicles (UAVs, UGVs). UAVs are tasked with long-term surveillance and reactivity to events over a wide area, but are subject to energy constraints limiting their operation. UGVs can support UAVs by replenishing their battery via docking. The presented problem is a generalization of well-known NP-Hard problems in vehicle routing. We present a multi-level framework that decouples the challenging problem into smaller sub-problems for solving. The resulting framework is one that interleaves planning and execution for a team of vehicles, allowing for long-term operation and reactivity. We also present simulation results to validate our proposed approach.

Keywords: Autonomous Vehicle Navigation, Motion and Path Planning, Computer Vision for Transportation
Abstract: This paper presents a global localization solution for a stationary robot in an airport environment using a probabilistic particle filter solution enhanced with scan matching, showing significant improvements in accuracy and convergence time. Previous work uses adaptive Monte Carlo localization to provide a global pose estimate against airplanes of known geometry based on observations from a multi-beam lidar. This solution converges quickly but can sometimes lead to incorrect poses due to the symmetric geometry of the airplane and the sparse observation data obtained when stationary. By integrating the particle filter with a scan matching algorithm that registers the point cloud against a known model of the airplane, the convergence rate, accuracy, and computation time of the localization solution is greatly improved. Additionally, the computed cloud to mesh distance allows the system to reliably detect if the obtained localization result is correct. The solution has been implemented within the Robot Operating System (ROS) framework and can run in real-time on a low-power computing device on the robot. Comparison of the improvements are shown in simulation and the system is validated in practice on several airplanes at two airports showing superior performance.

Keywords: Collision Avoidance, Autonomous Agents, Autonomous Vehicle Navigation
Abstract: A priority tracking framework to enable scalable tracking of pedestrians for self-driving cars in dense scenes is developed. Reachability analysis is used on the ego-vehicle and pedestrian tracks to assign different tracking strategies based on perceived priority. Therefore, computationally heavy algorithms such as 3D multi-object tracking (MOT) can be performed on priority objects, while less relevant tracks are maintained by lower-level trackers. The approach is empirically evaluated in simulated and real traffic scenarios with dense detection of pedestrians. We show that our approach reduces the number of objects tracked by the 3D MOT by 5 fold when compared to the standard approach, and ensures the ego vehicle satisfies the same key criteria for passive safety.

Keywords: Robust/Adaptive Control, Autonomous Vehicle Navigation, Autonomous Agents
Abstract: This paper presents the design of a flatness based linearisation control approach for the longitudinal and lateral dynamics of an autonomous ground vehicle. Since the system dynamics can be affected by unavoidable modelling uncertainties and disturbances, this motivates the introduction of sliding mode controllers to further robustify the proposed control scheme. More precisely, relying on a 3 degrees-of-freedom (DoF) nonlinear single-track model of the vehicle, its flatness properties are analysed and a state-feedback linearisation is then applied in order to transform the nonlinear vehicle model into a Brunovsky canonical form, which is eligible for designing sliding mode controllers of suitable order. Finally, simulation results, carried out on a realistic vehicle model, are illustrated to assess the proposal even in comparison with a classical proportional-integral-derivative (PID) control law.

Keywords: Autonomous Agents, Reinforcement, Robust/Adaptive Control
Abstract: In this paper we deal with the control of platoon of autonomous vehicles driving in Urban Road Networks (URNs). We exploit the idea of using Deep Reinforcement Learning (DRL) as the path planner of the proposed architecture. The advantage of this solution is the capability to deal with the actual traffic congestion while driving the autonomous vehicle to its destination. In particular, the high-level routing decisions are translated into manipulable set-points for Receding Horizon controllers by making more computational affordable and efficient the control action on the vehicle dynamics. Feasibility and asymptotic closed-loop stability are formally proved. Some simulations on a platoon, consisting of three agents described by double-integrator models, are provided to show the effectiveness of the overall architecture.

Keywords: Intelligent Transportation Systems
Abstract: A low-cost 2D automotive radar is often used in autonomous driving and advanced driver-assistance systems. However, the 2D radar often assumes all detected objects to be on the ground plane when estimating radar/vehicle ego-velocity. However, when there are elevated background objects in presence, such as buildings and tall trees, the ego-velocity estimation tend to be biased. Here we analyze the source of estimation error and develop a new algorithm to recognize three types of object reflections using the discrepancy between the estimated ego-velocity and the measured Doppler velocity. We propose an elevation and background aware cost (EBAC) function to formulate an optimization framework which can distinguish the object types to improve ego-velocity estimation. We combine a robust estimation method with the optimization framework to handle outliers in radar readings. We have implemented the algorithm and tested it in both simulation and physical experiments using our autonomous vehicle. The results show that our estimation method significantly reduces ego-velocity estimation error while maintaining a smaller error variance without losing robustness. More specifically, it reduces the ego-velocity estimation error by 49% in the most common driving scenario.

Keywords: Agricultural Automation, Autonomous Vehicle Navigation, Sensor-based Control
Abstract: For agriculture to become more sustainable, new practices and new cropping systems are needed to limit inputs such as fertilisers and phytosanitary products. This paper exposes an autonomous vision-based approach for navigation and intra-row weeding. To preserve crops (here concerned with maize and bean), a plant classification algorithm using easy-to-extract features has been developed and integrated on a real agricultural system. The approach uses frontal and proximal detection; the first allows for autonomous vision-based navigation without relying on GPS; and the latter regulates the weeding tool action. Realistic simulation scenarios are presented in order to validate the proposed approach as well as open-field experiments in real agricultural conditions obtained within the framework and evaluation campaigns of the ROSE challenge.

Keywords: Computer Vision for Automation
Abstract: In this paper, we studied HueCode, i.e., a meta-marker with multiple fiducial markers overlaid in different colored layers. HueCode consists of relative pose markers (e.g., AR markers) and additional information markers (e.g., QR codes) through overlaying within the area of a single marker. Robots can simultaneously recognize the relative pose and additional information required for movement and recognition from a single HueCode. However, conventional coloring schemes and recognition methods deal with the recognition performance being degraded by illumination. Thus, we propose HueCode2 to solve this problem. The new coloring scheme allows for the use of any color that is easy to distinguish. The new recognition method uses SVMs trained under various illumination conditions to identify colors. The experiments showed that these methods improved recognition rates over conventional HueCode under various illumination conditions. Additionally, we propose AR-HueCode, i.e., a meta-marker with many overlaid AR markers and a 6-DoF pose estimation method. In comparison to the conventional single AR marker, we confirmed that AR-HueCode could be recognized with a wider range, higher accuracy, and robustness to occlusions. Lastly, we propose Affordable-HueCode, which is optimized for inexpensive paper and printers with poor color reproduction.

Keywords: Computer Vision in Automation, Learning and Adaptive Systems, Machine learning
Abstract: This work presents a method for multiview object recognition and the estimation of the sequence of object views. The main feature of the method is to be lightweighted, which can be crucial for future sensor technology. This goal is achieved with a hidden Markov model of the 3D object from information of a sequence of 2D images. COIL 100 dataset were used for testing the method. The tests included images affected by rotation as well as Gaussian noise. Consistent results were obtained using relatively few shots of each class for the training process.

Keywords: Computer Vision for Automation, Deep Learning Methods
Abstract: The usefulness of deep learning models in robotics is largely dependent on the availability of training data. Manual annotation of training data is often infeasible. Synthetic data is a viable alternative, but suffers from domain gap. We propose a multi-step method to obtain training data without manual annotation effort: From 3D object meshes, we generate images using a modern synthesis pipeline. We utilize a state-of-the-art image-to-image translation method to adapt the synthetic images to the real domain, minimizing the domain gap in a learned manner. The translation network is trained from unpaired images, i.e. just requires an un-annotated collection of real images. The generated and refined images can then be used to train deep learning models for a particular task. We also propose and evaluate extensions to the translation method that further increase performance, such as patch-based training, which shortens training time and increases global consistency. We evaluate our method and demonstrate its effectiveness on two robotic datasets. We finally give insight into the learned refinement operations.

Keywords: Computer Vision for Automation, Deep Learning Methods, Big Data in Robotics and Automation
Abstract: We present a continuous perception approach that learns geometric and physical similarities between garments by continuously observing a garment while a robot picks it up from a table. The aim is to capture and encode geometric and physical characteristics of a garment into a manifold where a decision can be carried out, such as predicting the garment's shape class and its visually perceived weight. Our approach features an early stop strategy, which means that a robot does not need to observe a full video sequence of a garment being picked up from a crumpled to a hanging state to make a prediction, taking 8 seconds in average to classify garment shapes. In our experiments, we find that our approach achieves prediction accuracies of 93% for shape classification and 98.5% for predicting weights and advances state-of-art approaches in similar robotic perception tasks by 22% for shape

Keywords: Computer Vision for Automation, Logistics, Industrial Robots
Abstract: We developed parameterized B-rep-based surface correspondence estimation (PABSCO), which is a new category-level 3D object matching method to estimate the 6DoF poses and sizes of items consisting of multiple components with high recognition accuracy for practical use in logistics warehouses. PABSCO represents differences in the part sizes of each item belonging to a category by using parameterized boundary representation (B-rep), which is composed of multiple surface connections and parameters that control the sizes and positions of surfaces. It enables efficient, accurate estimation of the 6DoF pose and size of an unseen object instance by repeating part size adjustment and the surface registration between the model and areas segmented from scene data. In an experiment based on recognition for picking items in a logistics warehouse, on average, PABSCO achieved a high recognition rate of over 99.9% and a relative part size estimation error of less than 6.3% for both single- and multi-part logistics items.

Keywords: Computer Vision in Automation, AI-Based Methods, Data fusion
Abstract: Many outdoor autonomous mobile platforms require more human identity anonymized data to power their data-driven algorithms. The human identity anonymization should be robust so that less manual intervention is needed, which remains a challenge for current face detection and anonymization systems. In this paper, we propose to use the skeleton generated from the state-of-the-art human pose estimation model to help localize human heads. We develop criteria to evaluate the performance and compare it with the face detection approach. We demonstrate that the proposed algorithm can reduce missed faces and thus better protect the identity information for the pedestrians. We also develop a confidence-based fusion method to further improve the performance.

Keywords: Human Factors and Human-in-the-Loop, Human-Centered Automation, Task Planning
Abstract: The continued development of robots has enabled their wider usage in human surroundings. Robots are more trusted to make increasingly important decisions with potentially critical outcomes. Therefore, it is essential to consider the ethical principles under which robots operate. In this paper we examine how contrastive and non-contrastive explanations can be used in understanding the ethics of robot action plans. We build upon an existing ethical framework to allow users to make suggestions about plans and receive automatically generated contrastive explanations. Results of a user study indicate that the generated explanations help humans to understand the ethical principles that underlie a robot's plan.

Keywords: Human Factors and Human-in-the-Loop, Logistics, Virtual Reality and Interfaces
Abstract: We introduce an intuitive pointing-based interface to select objects moving on a system of conveyor belts. The interface has minimal sensing requirements, as the operator only needs to wear an Inertial Measurement Unit on the wrist (e.g., a smartwatch). LED strips provide the required visual feedback to precisely point to the objects and select them. We test the proposed approach in three environments of different complexity. Experiments compare our approach with a graphical interface where the user clicks on packages with a mouse; quantitative results show that our interface compares favorably, especially in difficult scenarios involving many packages moving fast.

Keywords: Human Factors and Human-in-the-Loop, Robotics and Automation in Life Sciences, Human-Centered Automation
Abstract: Teleoperation of robotic systems for precise and delicate object grasping requires high-fidelity haptic feedback to obtain comprehensive real-time information about the grasp. In such cases, the most common approach is to use kinesthetic feedback. However, a single contact point information is insufficient to detect the dynamically changing shape of soft objects. This paper proposes a novel telemanipulation system that provides kinesthetic and cutaneous stimuli to the user's hand to achieve accurate liquid dispensing by dexterously manipulating the deformable object (i.e., pipette). The experimental results revealed that the proposed approach to provide the user with multimodal haptic feedback considerably improves the quality of dosing with a remote pipette. Compared with pure visual feedback, the relative dosing error decreased by 66% and task execution time decreased by 18% when users manipulated the deformable pipette with a multimodal haptic interface in combination with visual feedback. The proposed technology can be potentially implemented in delicate dosing procedures during the antibody tests for COVID-19, chemical experiments, operation with organic materials, and telesurgery.

Keywords: Human-Centered Automation, Human Factors and Human-in-the-Loop, Design and Human Factors
Abstract: LinkGlide-S is a novel wearable hand-worn tactile display to deliver multi-contact and multi-modal stimuli at the user's palm.} The array of inverted five-bar linkages generates three independent contact points to cover the whole palm area. textcolor{black} {The independent contact points generate various tactile patterns at the user's hand, providing multi-contact tactile feedback. An impedance control delivers the stiffness of objects according to different parameters. Three experiments were performed to evaluate the perception of patterns, investigate the realistic perception of object interaction in Virtual Reality, and assess the users' softness perception by the impedance control. The experimental results revealed a high recognition rate for the generated patterns. These results confirm that the performance of LinkGlide-S is adequate to detect and manipulate virtual objects with different stiffness. This novel haptic device can potentially achieve a highly immersive VR experience and more interactive applications during telemanipulation

Keywords: Learning and Adaptive Systems, Autonomous Agents, Human Factors and Human-in-the-Loop
Abstract: In this paper an adaptive shared control framework for human agent collaboration is introduced. In this framework the agent predicts the human intention with a confidence factor that also serves as the control blending parameter, that is used to combine the human and agent control commands to drive a robot or a manipulator. While performing a given task, the blending parameter is dynamically updated as the result of the interplay between human and agent control. In a scenario where additional trajectories need to be taught to the agent, either new human demonstrations can be generated and given to the learning system, or alternatively the aforementioned shared control system can be used to generate new demonstrations. The simulation study conducted in this study shows that the latter approach is more beneficial. The latter approach creates improved collaboration between the human and the agent, by decreasing the human effort and increasing the compatibility of the human and agent control commands.

Keywords: Human-Centered Automation, Collaborative Robots in Manufacturing
Abstract: Diversity and inclusion in the industries is a new and challenging problem. Due to declining birthrates and aging populations, it is becoming difficult to recruit workers at various industrial sites. Automation is one solution, but technical limitations make it difficult to automate all industries entirely. Therefore, it is expected that more diverse people, such as the elderly people and people with poor skills for necessary work, will participate in various industries.

Human-robot collaboration (HRC) is an approach that can balance the loads of humans and machines while allowing for their limitations. If robots can understand human workloads and work capacity sufficiently, it will be possible to collaborate while employing "diversified people" and maintaining productivity.

This paper presents a system to manage time pressure in human-robot collaboration (HRC) systems to control human workload. To adjust time pressure using a real robot, the task scheduler to decide intervention timing of the robot and interaction system is proposed.

Our system was evaluated through a subjective experiment. In the experiment, workload and productivity were estimated using a physiological signal, a subjective evaluation and operation time.Based on the results of the experiment, we investigated the relation between workload and efficacy under the conditions of several time pressures. The result shows the proposed HRC system can control time pressure and the time pressure can affect human workload and productivity.

Keywords: Motion and Path Planning
Abstract: Path planning seeks to enable a robot to reach its goal while avoiding collisions with obstacles. Machine learning has been leveraged in recent years to improve the efficiency of path-planning approaches, particularly those based on Rapidly-exploring Random Tree (RRT). The general idea is to train a neural network using an extensive dataset consisting of solutions obtained by RRT for problems generated at random, and then leveraging the trained neural network to improve the sampling and exploration conducted by RRT on new problems.

This paper makes the case that the time to generate the datasets can be drastically reduced by using probabilistic roadmaps (PRM) instead of RRT. Roadmaps make it possible to capture the connectivity of the environment and solve multiple path-planning problems efficiently. We use the datasets to train neural networks based on U-NET. The trained networks are then leveraged to guide exploration. Specifically, we develop two path planners, namely Neural-Network-Guided PRM and RRT, denoted as NNG-PRM and NNG-RRT. Experiments are conducted using a series of challenging obstacle-rich environments where the robot has to wiggle its way through numerous narrow passages. The results show that using PRM instead of RRT drastically reduces the time to generate the datasets from hours to minutes. When comparing the planners on new problem instances, results show that NNG-PRM is faster than PRM, and NNG-RRT significantly outperforms RRT.

Keywords: Motion and Path Planning, Autonomous Agents, Reinforcement
Abstract: Underwater target localization using range-only and single-beacon (ROSB) techniques with autonomous vehicles has been used recently to improve the limitations of more complex methods, such as long baseline and ultra-short baseline systems. Nonetheless, in ROSB target localization methods, the trajectory of the tracking vehicle near the localized target plays an important role in obtaining the best accuracy of the predicted target position. Here, we investigate a Reinforcement Learning (RL) approach to find the optimal path that an autonomous vehicle should follow in order to increase and optimize the overall accuracy of the predicted target localization, while reducing time and power consumption. To accomplish this objective, different experimental tests have been designed using state-of-the-art deep RL algorithms. Our study also compares the results obtained with the analytical Fisher information matrix approach used in previous studies. The results revealed that the policy learned by the RL agent outperforms trajectories based on these analytical solutions, e.g. the median predicted error at the beginning of the target's localisation is 17% less. These findings suggest that using deep RL for localizing acoustic targets could be successfully applied to in-water applications that include tracking of acoustically tagged marine animals by autonomous underwater vehicles. This is envisioned as a first necessary step to validate the use of RL to tackle such problems, which could be used later on in a more complex scenarios.

Keywords: Motion and Path Planning, Autonomous Vehicle Navigation, Probability and Statistical Methods
Abstract: In the past decade, there is an increasing interest in the deployment of unmanned surface vehicles (USVs) for undertaking ocean missions in dynamic, complex maritime environments. The success of these missions largely relies on motion planning algorithms that can generate optimal navigational trajectories to guide a USV. Apart from minimising the distance of a path, when deployed a USVs’ motion planning algorithms also need to consider other constraints such as energy consumption, the affected of ocean currents as well as the fast collision avoidance capability. In this paper, we propose a new algorithm named anisotropic GPMP2 to revolutionise motion planning for USVs based upon the fundamentals of GP (Gaussian process) motion planning (GPMP, or its updated version GPMP2). Firstly, we integrated the anisotropy into GPMP2 to make the generated trajectories follow ocean currents where necessary to reduce energy consumption on resisting ocean currents. Secondly, to further improve the computational speed and trajectory quality, a dynamic fast GP interpolation is integrated in the algorithm. Finally, the new algorithm has been validated on a WAM-V 20 USV in a ROS environment to show the practicability of anisotropic GPMP2.

Keywords: Motion and Path Planning, Collision Avoidance, Optimization and Optimal Control
Abstract: We evaluate a centralized vs. a distributed approach for online trajectory generation for a fleet of mobile robots in the presence of both static and dynamic obstacles. The trajectories are generated online due to dynamic obstacles and this is formulated as a nonlinear model predictive control problem. In evaluating the centralized vs. distributed approach, the decentralized approach is shown to scale to many robots. In contrast, the computational cost of the centralized approach increases with the number of robots. When comparing the generated trajectories, the trajectories generated by the distributed control approach have larger deviations from a statically computed reference trajectory than those generated by the centralized approach. However, the evaluation shows that the distributed approach might be a viable alternative when the number of robots is large.

Keywords: Software Architecture for Robotic and Automation, Automation Technologies for Smart Cities
Abstract: When robots move through social spaces (i.e., environments shared with people) such as museums and shopping centers, they must navigate in a safe and socially acceptable manner to facilitate their inclusion and adoption. Therefore, robots operating in such settings must be able not only to avoid colliding with nearby obstacles, but also to show socially accepted behaviors, e.g., by minimizing the disruption in the comfort zone of nearby people. While there are well known approaches for social robot navigation, they are mostly based on social force models, which suffer from local minima. Meanwhile, other robot navigation frameworks do not consider social aspects. In this paper, we present an online social robot navigation framework, which is capable of generating collision free and socially acceptable paths online in uncontrolled crowded environments. Our proposed framework employs a modified sampling-based planner together with a new social relevance validity checking strategy. To evaluate our approach, we have designed a simulated social space in which the Pepper robot can safely navigate in a socially accepted manner. We compare our approach with other two alternative solutions while measuring specific social navigation metrics.

Keywords: Motion and Path Planning, Deep Learning in Robotics and Automation
Abstract: Fast generation of optimized robot motions is crucial for achieving fluent cooperation in shared workspaces. Established sampling-based motion planning algorithms are guaranteed to converge to an optimal solution but often deliver low-quality initial results. To this end, learning-based methods reduce planning time delays and increase motion quality. Existing methods show promising results for low-dimensional and simulated problems. In the real world, sensor noise or a change of the robot's tool can cause a distributional shift to the training data. An adaptive sampling strategy is thus required to cope with possibly suboptimal samples and ensure fast motion planning in human-robot collaboration. In this work, we present a sampling strategy for fast and efficient manipulator motion planning which is based on a conditional variational autoencoder. We test our model for three optimization objectives: path length in configuration space and workspace, as well as joint limit distances. In contrast to other works, we not only condition our model on the planning problem but also on motion progress. This allows for generating samples in the growth direction of the tree. Using our method, we obtain high-quality initial paths within less than one second of planning time.

Keywords: Optimization and Optimal Control, Sensor-based Control
Abstract: This paper proposes a control framework for changing the shape of linear deformable objects (LDOs) with a robotic arm without knowing the object's properties on a 2D workspace. In particular, we aim to provide a complete methodology that can be used in the future to manipulate agricultural LDOs, such as branches and twigs of vegetables, without damaging them. The first component of our framework is a shape prediction optimal method that obtains a target shape that minimizes the stress along the target's length. Using this method, the reachability of the target shape can be guaranteed. The second component of our framework is executed later and is based on an indirect optimal controller that automatically drives the objects' shapes into the target shapes by minimizing a cost function that reduces the error between the targets and the current shapes. To find the relation between the motion of the robotic arm and the object's shape, a Jacobian matrix is calculated by using the As-Rigid-As-Possible deformation model. Several numerical simulations and real experiments are presented to highlight the performance of the proposed methodology.

Keywords: Optimization and Optimal Control, Swarms
Abstract: The main contribution of this work is to solve the optimization problem associated to the gain tuning of a state feedback controller using a PSO algorithm through a new cost function. The latter is built by adding a term to the classic quadratic cost function used in the Linear Quadratic Regulator (LQR) method, which depends on the quadratic value of the time derivative of the control signal. The added term weights the amount of chattering, and the sharpness and level of the peaks appearing in the control signal. High levels of chattering may produce undesired oscillations and actuator wear, and reducing this behavior is paramount in practice. On the other hand, large peaks could generate actuator saturation which may lead to instability. An experimental comparative study is made between the proposed and the standard LQR method by means of an experimental platform. It is observed that the closed-loop system with the gains obtained through the proposed method produces a control signal with reduced peaks and less chattering compared with the signal obtained with the gain generated by the LQR method.

Keywords: Robust/Adaptive Control, Robust Manufacturing
Abstract: In the field of ink-jet deposition, there is a lack of specific knowledge to detect and change drop volume to regulate fluid placement in real time. We first derive the model of line width as a function of nozzle velocity, valve duty cycle, and physical properties of fluid and surface. We then propose a novel approach for simultaneous parameter estimation and tracking control to estimate unknown parameters while regulating line width in ink-jet deposition. Stability of both the estimator and control are established using Lyapunov stability theory. Simulations and experimental results confirm the stability and performance of the approach.

Keywords: Swarms, Haptics and Haptic Interfaces, Motion Control
Abstract: We present a game benchmark for testing human-swarm control algorithms and interfaces in a real-time, high-cadence scenario. Our benchmark consists of a swarm vs. swarm game in a virtual ROS environment in which the goal of the game is to “capture” all agents from the opposing swarm; the game’s high-cadence is a result of the capture rules, which cause agent team sizes to fluctuate rapidly. These rules require players to consider both the number of agents currently at their disposal and the behavior of their opponent’s swarm when they plan actions. We demonstrate our game benchmark with a default human-swarm control system that enables a player to interact with their swarm through a high-level touchscreen interface. The touchscreen interface transforms player gestures into swarm control commands via a low-level decentralized ergodic control framework. We compare our default human-swarm control system to a flocking-based control system, and discuss traits that are crucial for swarm control algorithms and interfaces operating in real-time, high-cadence scenarios like our game benchmark. Our game benchmark code is available on Github; more information can be found at https: //sites.google.com/view/swarm-game-benchmark.

Keywords: Compliance and Impedance Control, Industrial and Service Robotics
Abstract: Smooth behaviors are preferable for many contact-rich manipulation tasks. Impedance control provides an effective way to regulate robot movements by mimicking motions of a mass-spring-damping system. Consequently, the robot behavior can be determined by the impedance gains. However, tuning of the impedance gains for different tasks is not straightforward, especially for unstructured environments. Moreover, online adaption of the optimal gains to deal with the time-varying performance index is even more challenging. In this paper, we present Safe Online Gain Optimization for Cartesian space Variable Impedance Control (Safe OnGO-VIC) to overcome these challenges. By reformulating the dynamics of impedance control as a control-affine system, in which the impedance gains are the inputs, we provide a novel perspective to understand the relation between impedance gains and the robot behaviors. Furthermore, we innovatively formulate an optimization problem that utilizes force measurement collected online to obtain the optimal impedance gains in real-time. Safety constraints are also embedded in the proposed framework. We experimentally validated the proposed algorithm on three contact-rich manipulation tasks. Comparison results with a constant gain baseline and an adaptive control baseline prove that the proposed algorithm is effective and generalizable to different scenarios.

Keywords: Foundations of Automation
Abstract: In this paper, a bilateral teleoperation-based robot skill learning framework is developed to transfer multi-step and contact manipulation skills from humans to robots. Robot skill acquistion via bilateral teleoperation provides a solution for human teacher to transfer the manipulation skills to robots in a remotely feasible manner. Besides, the bilateral teleoperation with force feedback allows humans in the loop to monitor and interface with the robot behaviour, hence improving the safety of the robot execution. The dynamic movement primitive (DMP) model is first employed to encode primitive skills, including those for both the translation and orientation. We have been utilized the behaviour tree (BT) to model the sequence of primitive skills. Since each node of the BT represents a single primitive skill, we can reproduce the BT nodes by employing different controllers based on the task requirements. We have evaluated the approach through two robot manipulation tasks, (i) grasping irregular objects with a customized soft suction cup and (ii) wiping whiteboard by a 7-DoF Frank Emika Panda. Results and performance analysis of the experiments are presented subsequently.

Keywords: Semiconductor Manufacturing, Planning, Scheduling and Coordination, Petri Nets for Automation Control
Abstract: We consider a scheduling problem of a cluster tool, which is widely used for semiconductor manufacturing processes. A timed Petri net (TPN) is used to model the problem. We propose a new mathematical model based on a TPN by using the assignment problem and a branch-and-price approach.

Keywords: Semiconductor Manufacturing, Deep Learning Methods, Factory Automation
Abstract: With the development of deep learning, data-driven approaches have shown great success in the semiconductor manufacturing. For example, wafer bin maps (WBM) recognition is a critical application to identify failure modes and finding root-cause to reduce yield loss. However, the WBM studies provide static classification results without tracing the temporal patterns. This study develops a spatio-temporal strip map prediction system for the flip-chip bonding process in the assembly house. The proposed strip bin map (SBM) prognostic system including prediction model and recognition model can provide pre-alarm of defect and predict the defect mode for the final process. In practice, not all processes are followed by a functional test (FT) and thus this proposed system can simulate the defect generation based on the Bayes' theorem for tracking the changes of the spatio-temporal patterns. An empirical study of the semiconductor assembly manufacturer in Taiwan was conducted to validate the proposed framework, and the results show that the proposed SBM system successfully predict the SBM and defect mode for the final test and provide an effective pre-alarm.

Keywords: Reinforcement, Learning and Adaptive Systems, Deep Learning in Robotics and Automation
Abstract: Automated material handling systems (AMHSs) play a critical role in semiconductor fabrication plants (fabs). The primary type of AMHS used in fabs is the overhead hoist transport (OHT) system, which transports lots between processing machines. A modern large-scale fab may operate thousands of OHT vehicles and thus often experiences OHT vehicle congestion. This paper proposes a reinforcement learning-based dynamic routing algorithm to address the OHT vehicle congestion problem, and develops a graph neural network–based predictive model to determine in advance the situation on an OHT vehicle’s succeeding track. This predictive model enables the algorithm to function well, regardless of the distribution of data and the topology of the track. We show via simulation that this novel algorithm reduces the mean OHT vehicle travel time in a controlled case. In future work, we will conduct simulation verification and then apply our model to a commercial OHT management system to study its real-world in-fab performance.

Keywords: Factory Automation, Intelligent and Flexible Manufacturing, Collaborative Robots in Manufacturing
Abstract: We present a dynamic programming-based heuristic algorithm for selecting the next task of in-process cars in automotive assembly plants. As car production demand diversifies, manufacturing processes require high flexibility. An alternative to a conventional production line, matrix-structured production system provides a flexible manufacturing environment. In this system, until the final assembly operation is completed, each autonomous mobile robot (AMR) is assigned to a single car body and transports among various workstations to accomplish its given operations. These operations require sequential procedure and can be performed at multiple workstations. Therefore, assigning AMRs to appropriate workstations and operations is the key decision making that determines the capacity of the plants. In this paper, we formulate the Markov decision process (MDP) model for the flexible job shop scheduling problem with a single AMR and get the optimal policy by applying dynamic programming. Moreover, we suggest a heuristic algorithm to deal with multiple AMRs. We, then, validate the proposed algorithm by comparing various algorithms using simulation. Finally, we analyze the effect of the scheduling algorithm on production capacity.

Keywords: Reinforcement Learning, Intelligent and Flexible Manufacturing, Factory Automation
Abstract: As industry rapidly shifts towards mass personalisation, the need for a decentralised multi-agent system capable of dynamic flexible job shop scheduling (FJSP) is evident. Traditional heuristic and meta-heuristic scheduling methods cannot achieve satisfactory results and have limited application to static environments. Recent Reinforcement Learning (RL) approaches that consider dynamic FJSP, lack flexibility and autonomy as they use a single-agent centralised model, assuming global observability. As such, we propose a Multi-Agent Reinforcement Learning (MARL) system for scheduling dynamically arriving assembly jobs in a robot assembly cell. We applied a Double DQN-based algorithm and proposed a generalised observation, action and reward design for the dynamic FJSP setting. Using a centralised training phase, each agent (i.e., robot) in the assembly cell executes decentralised scheduling decisions based on local observations. Our solution demonstrated improved performance against rule-based heuristic methods, for optimising makespan. We also reported the impact of different observation sizes of each agent on optimisation performance.

Keywords: Health Care Management, Agent-Based Systems, Robotics and Automation in Life Sciences
Abstract: The COVID-19 pandemic has exposed long-standing deficiencies in critical care knowledge and practice in hospitals worldwide. New methods and strategies to facilitate timely and accurate interventions are needed. A virtual counterpart (digital twin) to critically ill patients would allow bedside providers to visualize how the organ systems interact to cause a clinical effect, offering them the opportunity to evaluate the effect of a specific intervention on a virtual patient before exposing an actual patient to potential harm. This work aims at developing a digital simulation that models the clinical pathway of critically ill patients. Using the mixed-methods approach with the support of multiprofessional clinical experts, we first identify the causal and associative relationships between organ systems, medical conditions, clinical markers, and interventions. We record these relationships as structured expert rules, depict them in a directed acyclic graph (DAG) format, and store them in a graph database (Neo4j). These structured expert rules are subsequently utilized to drive a simulation application that enables users to simulate the state trajectory of critically ill patients over a given simulated time period to test the impact of different interventions on patient outcomes. This simulation model will be the engine driving a future digital twin prototype, which will be used as an educational tool for medical students, and as a bedside decision support tool to enable clinicians to make faster and more informed treatment decisions.

Keywords: Health Care Management, Modelling, Simulation and Optimization in Healthcare, Clinical and Operational Decision Support
Abstract: To improve care quality and efficiency of proton therapy systems, the facility has options to upgrade proton accelerator. However, operating a proton therapy facility is complex, and thus it is not obvious to see if an upgrade is necessary and to what level a facility should be upgraded to better serve patients. This study aims to explore the relationship between accelerator upgrade and system level performance. The time of each proton therapy procedure was collected from the proton therapy practice at Mayo Clinic and applied to a simulation model. The current state of 4-gantry proton therapy system was taken as a baseline, and different upgrade options were evaluated. It suggested that a small dose rate increase could achieve a significant system improvement in terms of patient throughput and beam wait time. Besides, the necessity of accelerator upgrade for 3-gantry and 2-gantry proton therapy systems was studied. The improvement of accelerator upgrade for systems with less gantries was mainly demonstrated by decrease of finish time. This study can help a proton beam facility make a proper upgrade decision.

Keywords: Scheduling in Healthcare, Health Care Management, Modelling, Simulation and Optimization in Healthcare
Abstract: This paper addresses the dynamic scheduling of pre-surgery appointments at different medical services needed for hip and knee replacement. The key concerns are the minimization of the number of hospital visits and having all appointments before the surgery. This paper proposes a multistage approach including (i) design of multi-appointment patterns, (ii) assignment of multi-appointments to different patient types (iii) dynamic appointment scheduling. The proposed approach is tested on a real-life case study on pre-surgery appointments for hip and knee replacement in a healthcare institution in Saint-Etienne, France and is compared to some others simple benchmarking policies.

Keywords: Deep Learning Methods, Health Care Management, Probability and Statistical Methods
Abstract: As a pervasive issue, missing data may influence the data modeling performance and lead to more difficulties of completing the desired tasks. Many approaches have been developed for missing data imputation. Recently, by taking advantage of the emerging generative adversarial network (GAN), an effective missing data imputation approach termed generative adversarial imputation nets (GAIN) was developed. However, its modeling architecture may still lead to significant imputation bias. In addition, with the GAN structure, the training process of GAIN may be instable and the imputation variation may be high. Hence, to address these two limitations, the ensemble GAIN with selective multi-generator (ESM-GAIN) is proposed to improve the imputation accuracy and robustness. The contributions of the proposed ESM-GAIN consist of two aspects: (1) a selective multi-generation framework is proposed to identify high-quality imputations; (2) an ensemble learning framework is incorporated for GAIN imputation to improve the imputation robustness. The effectiveness of the proposed ESM-GAIN is validated by both numerical simulation and two real-world breast cancer datasets.

Keywords: Clinical and Operational Decision Support, Machine learning, Data fusion
Abstract: Recently, clinical decision making processes largely benefit from electronic health records (EHR) to improve the diagnosis outcomes and quality of patient care. In particular, patients' conditions in critical care are routinely monitored through essential vital signs such as respiratory rate, heart rate and blood pressure, and lab results. These patient variables in a spatio-temporal and irregular form provide valuable information to support time-dependent decision making. However, the absence of values in these data is almost inevitable due to various causes including irregular sampling rate, sensor drop-off, or manual interference during the data collection process. It, therefore, inhibits the extraction of meaningful information using data-driven machine-learning techniques. To this end, we propose a novel imputation method to enhance data quality. %the performance of various state-of-art data imputation methods. In our approach, we improve the performance of pre-filling methods including mean imputation and interpolations by 7%-95% on both synthetic dataset and MIMIC III database. Also, our method improves the mean imputation and outperforms the 3D-MICE for 9 out of 13 laboratory variables by 25%-80% on 8267 MIMIC III patients.

Keywords: Optimization and Optimal Control, Modelling, Simulation and Optimization in Healthcare
Abstract: The optimization problems that decision makers often face in practice are that they need to select satisfactory designs from alternatives to meet their needs. Due to the randomness, the performance of all designs needs to be evaluated via simulation. Since simulation is time-consuming and costly, how to improve the efficiency of simulation to finding the satisfactory designs within a limited simulation budget constitutes a ranking and selection problem. In the existing literature, the performance of each design in almost all ranking and selection problems can be evaluated individually. But in real production and life, there are many situations where the performance of design can only be evaluated by the results of pairwise comparison. For example, in the round-robin competition of the Winter Olympics curling competition, each participating team has to play against other teams once, and the ranking of teams is ranked according to the results of the paired competition. This paper considers the ranking and selection problem where the performances of designs cannot be evaluated individually, but only pairwise. Under this setting, we use the Borda score as a performance measure, and build an optimal computing budget allocation (OCBA) model for the goal of selecting the best design from finite alternatives within a limited number of simulation replications. Based on the model, we make some approximations, and get an asymptotically convex programming problem. We use the Karush–Kuhn–Tucker (KKT) conditions to solve the problem, and derive asymptotic optimality conditions, which guide a sequential algorithm for simulation budget allocation. The numerical testing shows that the proposed algorithm can effectively improve the simulation efficiency. We also make an extension to sele

Keywords: Modelling, Simulation and Validation of Cyber-physical Energy Systems, Power and Energy Systems automation, Distributed Generation and Storage
Abstract: This speech will discuss the new structure of green energy systems and how zero carbon emission energy system can be realized. In fact economic energy storage technology is the key for fully utilizing new renewable energy sources. Production, storage and transportation, and utilization of hydrogen as a main energy source are introduced in the speech, and it is shown that hydrogen can become a major secondary energy source as important as electricity. The hydrogen based intelligent energy system will provide a new solution for energy supply and consumption with nearly zero-carbon emission, and may lead to the energy revolution in the near future.

Keywords: Planning, Scheduling and Coordination
Abstract: Data analytics is the frontier basic research direction of industrial intelligence and one of the driving forces to promote scientific development. Systems optimization is the core basic theory of decision-making in smart industry, as well as the heart and engine of data analytics. This talk will discuss some systems modeling methods and optimization solution methods we have been working on. The systems modeling methods are to quantitatively describe different practical problems with proper formulations, including set-packing model, space-time network model, and continuous-time based model. The optimization solution methods include integer optimization, convex optimization, intelligent optimization, and dynamic optimization. This talk will also introduce systems optimization and data analytics of production, logistics, and energy in the steel industry, including: 1) production batching and scheduling in steelmaking/continuous casting, and hot/cold rolling operations; 2) logistics scheduling in loading operations, shuffling/reshuffling, and stowage; 3) data analytics-based energy optimization, including dynamic energy allocation and scheduling, energy analytics covering energy description, diagnosis and prediction; 4) data analytics, including temperature prediction of blast furnace, dynamic analytics of BOF steelmaking process based on multi-stage modeling, temperature prediction of reheat furnace based on mechanism and machine learning, and strip quality analytics of continuous annealing based on multi-objective ensemble learning.

Keywords: Deep Learning in Robotics and Automation, Brain-Machine Interfaces
Abstract: Visual localization of micro flexible electrode and implant needle is an important task for robotic flexible electrode implantation. Magnification switch, occlusion, defocus, illumination changes in microscopic imaging produce challenges for this task. We propose the Keypoint Localization and Angle Estimation Network (KLAE-Net) based on convolutional neural networks. KLAE-Net has two branches: the keypoint localization branch for obtaining the coordinates of electrode and needle in image space; the angle estimation branch for monitoring the inclination of needle. Attention mechanism and deformable convolution are used to improve the model’s performance. For training and evaluation under the flexible electrode implantation task, we construct a novel dataset containing 1000 images covering various conditions. An image Jacobian matrix based alignment control method is designed, to realize the robotic alignment between needle and electrode. A series of experiments are conducted with the dataset and an implantation robot system.

Keywords: Automation at Micro-Nano Scales, Biological Cell Manipulation, Computer Vision for Automation
Abstract: A self-recalibrating micromanipulator system is developed in this work to perform vision-based control under unpredictable scenes and poor imaging conditions. To ensure resilient vision-based control in such challenging conditions, we proposed a self-recalibrating mechanism that combines the estimates from both the robot manipulator and microscope vision. Our method is demonstrated for micropipette cell aspiration. In the initialization stage, the system automatically locates the tip and selects the templates for visual tracking. At the self-calibration phase, accurate tracking of the tip is achieved by the background subtraction template matching (BSTM) method in order to estimate the calibration matrix in complex environments. During the operation, the control error is detected according to the visual information and the self-recalibration is executed when the error exceeds the threshold value. Experimental analysis shows that the BSTM method for self-calibration achieves accurate tracking with an average error of less than 1 pixel (0.18 μm). And the system has an average control error of 4.18 pixels (0.75 μm) under complex calibration backgrounds, satisfying the accuracy requirements for cell aspiration. Moreover, the system achieved stable tracking in complex operating environments, including cell interaction, occlusion, tip leaving the focal plane or beyond the field-of-view. By designing a self-recalibrating micromanipulator equipped with resilient vision-based control system, we hope to robotize the procedures for the study of cell mechanics during micropipette aspiration and pave the path for the advancement of robotic micromanipulation in biomechanical research.

Keywords: Motion and Path Planning, Collision Avoidance, Industrial and Service Robotics
Abstract: To obtain a reliable collision-free path, relevant constraints can be added to the robot. In this paper, safety reinforcement learning with kinematic constraints and torque-limited is studied to ensure safety in planning, with the designing of reinforcement learning action space to ensure the feasibility of action. For evaluation, path planning was carried out in an industrial welding scenario to allow the robot to reach the welding point in the narrow space. The experimental results show that the proposed method not only ensures convergence but also ensures the safety and reliability of the task.

Keywords: Robot Networks, Agent-Based Systems, Industrial and Service Robotics
Abstract: As robots play an increasingly important role in people's lives, researchers are working on robotic vehicles with powerful intelligence. However, a problem that cannot be ignored is resource constraints on the edge. Considering the gaming issues of resource constraints and intelligence level, we focus on robots with limited computing resources and propose an idea of changing from lightweight to powerful intelligence for a smart robotic system. Firstly, we design a series of ultra-lightweight algorithms according to the lightweight resource Limitation. Second, we collaborate the ultra-lightweight algorithms through a centralized-distributed architecture to achieve intelligent upgrading of the whole system. Then, by maximizing the use of resources and information, we accomplish a heterogeneous ultra-lightweight multi-robotic collaborative system. Finally, the presented architecture has been applied to realize a lightweight simultaneous localization and mapping (SLAM) system. Experimentally, the ultra-lightweight algorithm achieves 900fps on the server experimental platform. Since there have been less heterogeneous collaborative methods, we further compared it with the state-of-the-art homogeneous collaborative system and proved that the accuracy of our proposed system was improved by 45.98%.

Keywords: Motion Control, Industrial and Service Robotics, Motion and Path Planning
Abstract: Abstract— The quasi-static gait is a common walking strategy for legged robots. It can make the legged robots adapt to many structured terrains. Many researchers focus on the quasi-static gait for Quadruped Robots which show great efficiency and simplicity. But when applied to biped robots, the stability analysis and the gait planning are still two challenging issues. This paper focuses on the quasi-static gait for biped walking. A novel gait with the sinusoidal movement of the center of gravity is proposed to achieve smooth and fast quasi-static walking. Based on the relationship of ZMP and COG, we proposed a stability criterion for quasi-static walking and a method that adjusts the parameters of the walking pattern to achieve fast walking. The proposed method is validated by simulation analysis in VREP software. The results show that biped robots can have well-performed on flat ground.

Keywords: Manipulation Planning, Reactive and Sensor-Based Planning, Semiconductor Manufacturing
Abstract: Nanomaterials possess superior mechanical, electrical, and optical properties suitable for device applications in different fields such as nanoelectronics, photonics, and sensors. Characterizing the multiphysical properties of single nanomaterials and nanostructures provides experimental guidelines for synthesis and device applications of functional nanomaterials. Nanomanipulation techniques under scanning electron microscopy (SEM) have enabled the testing of mechanical and electrical properties of various nanomaterials. However, the introduction of micro-photoluminescence (μ-PL) measurement into an SEM setup for in-situ single nanomaterial characterization is still experimentally challenging; in particular, the seamless integration of the mechanical, electrical, and μ-PL testing techniques inside an SEM for multi-field-coupled characterization of single nanostructures is still unexplored. In this work, we report the first SEM-based nanomanipulation system for multiphysical characterization of single nanomaterials. A custommade, optical-microfiber-based μ-PL setup is integrated onto a nanomanipulation system with four nanomanipulators inside an SEM. The system is also equipped with a conductive nanoprobe and a conductive atomic force microscopy (AFM) probe for electrical nanoprobing and electroluminescence (EL) measurement of single nanomaterials with contact force feedback. Using the system, field-coupled characterization (i.e., optomechanical,optoelectronic, electromechanical, and mechano-optoelectronic testing) of single InGaN/GaN nanowires (NWs) are conducted; and, for the first time, the effect of mechanical compression applied to individual InGaN/GaN NWs on its optoelectronic property is revealed.

Keywords: Computer Vision for Manufacturing, Computer Vision in Automation, AI-Based Methods
Abstract: Surface defect recognition plays a crucial role in ensuring product quality in manufacturing systems. Deep convolutional neural networks (CNNs) achieve excellent performance on surface defect recognition, which requires a large amount of data and roughly uniform distribution of class labels for training. However, in the real world, surface defect recognition of products typically exhibits defect irregularities and imbalanced class distribution, some of which have very few samples. This is a great challenge to the generalization of the model on such classes. Therefore, this paper proposes a logit adjusting transformer-based method (LAT) to address the class imbalance problem in surface defect recognition. LAT encourages a larger margin between logits of the minority class and the majority class to achieve accurate recognition of the minority class. The LAT achieves better results compared with competing methods on a dataset of printed circuit boards (PCB) surface defects with five classes collected from a real-life manufacturing factory. The accuracy reaches 93.06% on the dataset with only ten samples in the minority class, which is improved by 7.52% to the best competing method.

Keywords: Computer Vision for Manufacturing, Computer Vision in Automation, AI-Based Methods
Abstract: Image anomaly detection is an important stage for automatic visual inspection in intelligent manufacturing systems. The wide-ranging anomalies in images, such as various sizes, shapes, and colors, make automatic visual inspection challenging. Previous work on image anomaly detection has achieved significant advancements. However, there are still limitations in terms of detection performance and efficiency. In this paper, a novel Position Encoding enhanced Feature Mapping (PEFM) method is proposed to address the problem of image anomaly detection, detecting the anomalies by mapping a pair of pre-trained features embedded with position encodes. Experiment results show that the proposed PEFM achieves better performance and efficiency than the state-of-the-art methods on the MVTec AD dataset, an AUCROC of 98.30% and an AUCPRO of 95.52%, and achieves the AUCPRO of 94.0% on the MVTec 3D AD dataset.

Keywords: Computer Vision for Manufacturing, Factory Automation, Hybrid Strategy of Intelligent Manufacturing
Abstract: Grease injection is one of the most important parts of high-speed rail maintenance, and whether the bolts on related facilities contain anomaly plays a crucial role in the safety of the entire grease injection process. Bolt anomaly detection is typically performed by either mechanical engineers or certified inspectors. However, this task is not only hazardous for the personnel but also extremely time-consuming. The decisions that depend entirely on the individuals’ experiences are always subjective. Therefore, this article presents an efficient bolt anomaly detection framework based on a semi-supervised technique. We first train a novel U-Net-based neural network with anomaly-free samples, which can simultaneously predict the positions of all bolts and segment areas that contain anomaly-free bolts. Subsequently, we build a novel variance distribution model to extract the prior knowledge of areas that contain anomaly-free bolts based on the segmentation results and predicted position results from the network. Finally, we utilize the pre-built model to generate anomaly response maps of test images, which intuitively visualize the areas containing abnormal bolts. The experimental results demonstrate that our proposed bolt anomaly detection framework achieves satisfactory accuracy and efficiency, which can meet the requirements of practical engineering detection.

Keywords: Big-Data and Data Mining, Cyber-physical Production Systems and Industry 4.0, Probability and Statistical Methods
Abstract: Intrusion detection systems (IDSs) are widely deployed in the industrial control systems to protect network security. IDSs typically generate a huge number of alerts, which are time-consuming for system operators to process. Most of the alerts are individually insignificant false alarms. However, it is not the best solution to discard these alerts, as they can still provide useful information about network situation. Based on the study of characteristics of alerts in the industrial control systems, we adopt an enhanced method of exponentially weighted moving average (EWMA) control charts to help operators in processing alerts. We classify all detection signatures as regular and irregular according to their frequencies, set multiple control limits to detect anomalies, and monitor regular signatures for network security situational awareness. Extensive experiments have been performed using real-world alert data. Simulation results demonstrate that the proposed enhanced EWMA method can greatly reduce the volume of alerts to be processed while reserving significant abnormal information.

Keywords: Computer Vision for Manufacturing, Collaborative Robots in Manufacturing, Computer Vision in Automation
Abstract: 3D object appearance inspection plays an important role in manufacturing industry. To observe clear images of different parts of a 3D object in a semi-structured scene, camera pose should be properly adjusted to several different viewpoints. In this paper, we propose a flexible appearance observation framework for 3D-shaped objects with 3-DoF pose uncertainty. First, we propose PR3Net based on convolutional neural network (CNN), to estimate the 2D position and 1D angle of a target 3D object placed on a platform. Considering the data scarcity problem in practical application and the variety of object types, we utilize data synthesis to automatically generate training samples from only one annotated image sample, so that the pose learning can be conducted conveniently. Besides, a semi-supervised fine-tuning method is used to improve the generalization ability by leveraging plenty of unlabeled images. Second, the teachable active motion strategy is designed to enable the inspection robot to observe a 3D object from multiple viewpoints. The human user teaches the standard viewpoints once beforehand. The robot actively moves its camera multiple times according to both the predefined viewpoints and the regressed 3-DoF pose, so that the images of multiple parts of object are collected. The effectiveness of the proposed methods is validated by a series of experiments.

Keywords: Sustainable Production and Service Automation, Planning, Scheduling and Coordination
Abstract: Achieving sustainability of production scheduling is currently an important development direction for the manufacturing industry. In view of the absence of efficiently sustainable evaluation for production scheduling schemes, this letter proposes a novel multi-criteria evaluation method based on fuzzy analytic network process (FANP) and cumulative prospect theory-enhanced Vlsekriterijumska optimizacija IKOmpromisno Resenje (CPT-VIKOR). To this end, we firstly identify fundamental performance indicators from extensive literature, and construct a four-layer indicator system framework in light of the process sustainability index (ProcSI) architecture. Then, the weights of all indicators are derived by designing a fuzzy-improved ANP method to better quantitatively depict the expert preference. Thereafter, VIKOR combined with CPT is put forward to obtain the ranking of candidate scheduling solutions, in which CPT is capable to reflect the mental attitudes of a decision maker via the prospect value of a solution. A case study for the flexible job shop validates the correctness and effectiveness of the proposed methodology. Meanwhile, the stability and sensitivity are further discussed, suggesting good potential in real-world applications.

Keywords: Planning, Scheduling and Coordination, Optimization and Optimal Control, Intelligent Transportation Systems
Abstract: This paper considers a coordinated global shipment matching problem in which a global operator receives shipment requests from shippers and three local operators provide local transport services in different geographical areas. While local operators make local matching decisions, the global operator combines the matched local services into itineraries to provide integrated transport for shipments. To handle the interconnecting constraints between different operators, a Lagrangian relaxation heuristic approach is developed. Under the proposed approach, the original problem is decomposed into local operator-related subproblems. These subproblems are optimized iteratively under local constraints as well as under the incentives imposed by the global operator to meet interconnecting constraints. The experiment results show that with the proposed approach, global transport planning that requires coordination among different operators to achieve a common goal can be realized.

Keywords: Product Design, Development and Prototyping, Industrial and Service Robotics, Motion Control
Abstract: Omnidirectional vehicles (ODVs) have wide applications, but most of them (e.g., mobile robots with omni-, Mecanum, or spherical wheels) are mainly designed for indoor use on flat and smooth terrains. Literature review indicates that mobile robots based on the "self-sustained" active split offset caster (ASOC) module design that uses conventional wheels (e.g., rubber tires) is more suitable to execute agile maneuvers in unstructured rough terrains. However, these robots often have time delay and synchronization issues caused by the wireless transmission of control signals and the wheel-terrain contact-breaking issues (some wheels are lifted off from the ground), which often lead to poor motion control and trajectory tracking performance when executing high speed turns. To solve these problems, through improved ASOC module design, active suspension design, and control algorithm design, we develop an ASOC-based mobile robot capable of active body posture control and agile omnidirectional mobility. We give detailed explanations of its design philosophy and working principle. Experiment results indicate that our proposed robot can achieve much better performance in challenging tests such as negotiating uneven ground and executing very sharp turns at high speed.

Keywords: Motion Control, Robust/Adaptive Control, Medical Robots and Systems
Abstract: In this paper, a dual quaternion-based uncertainty and disturbance estimator (UDE) controller is proposed for a redundant robot manipulator. By employing dual quaternion, the geometric parameters (including position, orientation, velocity and acceleration) of robot manipulator motion are accurately represented without singularity. Then, the dynamics model of a 7-DOF redundant manipulator is established in dual quaternion space, which can describe the coupling effect between translation and rotation. The tracking error is also defined to guarantee that the posture of the end-effector can converge to the desired trajectory. With twist error dynamics, the UDE control framework is developed. This framework can improve the robust trajectory performance in the case of nonlinearity, model uncertainty and external disturbance. The dual equilibrium problem is also handled with the introduction of the parameter se, which represents the symbolic function of the dual quaternion error real part. Finally, the tracking performance and robustness of the proposed controller are validated by numerical simulations.

Keywords: Reinforcement, Big-Data and Data Mining, Machine learning
Abstract: Recommendation systems (RSs) play an important role in dealing with information overload, facilitating decision-making, and boosting business. For RSs, it is essential to capture the sequential dynamics of the user’s preference. Recently, reinforcement learning (RL) has been introduced into recommendation systems because of its advantages in considering dynamic interests. Nevertheless, the intricacy and sparsity of data make it uneasy to employ RL techniques in RSs. Intention-aware approaches capture the instinct driver of user behaviors and achieve considerable performance promotion. Inspired by that, we plan to study a novel intention-aware reinforcement learning model for the sequential recommendation. More specifically, the next-item recommendation task is formulated as a Markov decision process. State representation, reward function, and learning algorithm are specially designed for the intention-aware environment.

Keywords: Formal Methods in Robotics and Automation, Foundations of Automation
Abstract: Error model has a great effect on the performance of image-based visual servoing scheme, such as convergence rate, smoothness of control inputs (e.g. velocity commands), etc. For a long time, the classical first-order error model dominates the design of the visual servoing control law, and the second-order error model is not employed to design the control law until recent years, both of which guarantee that the error decreases exponentially. In this paper, by adding an adjustable exponential basis to the feature errors, a new error model for the image-based visual servoing scheme is proposed. To the best of our knowledge, this is the first time that such an error model is developed. Compared with the existing first-order error model and second-order error model, the proposed error model has comprehensive advantages in computational efficiency, convergence rate and depth errors robustness, although it has a disadvantage in velocity smoothness. Comparative numerical simulations and real experiments conducted on a six-axis industrial robot confirm the performance of the proposed error model.

Keywords: Optimization and Optimal Control, Cloud Computing For Automation, Agent-Based Systems
Abstract: The cloud-based multi-agent system (MAS) results from the convergence of cloud computing and MAS, and the trustworthy data, the data that best matches the personalized demands of agents on trust attributes of data, is important for the cloud-based MAS. Data usage often comes with cost, and the better the trustworthiness of data (TOD), the higher the cost of data usage (CDU). Therefore, how to make the optimal trade-off for cloud-based MAS between TOD and CDU arises as an interesting issue. To address this issue, this paper proposes a mathematical constrained optimization model to help the agents to obtain the trustworthy data within the acceptable level of CDU. Experiments on a public dataset show its effectiveness.