Thavornvong, Supakorn (Mahidol University), Kitsommart, Anapat (Mahidol University), Thabua, Mahannop (Mahidol University), Yuenyong, Sumeth (Mahidol University), Thanaluck, Chutiya (Mahidol University)

Voice-Controlled Robotic Arm System for Tabletop Manipulation Via Large Language Model and 3D Vision

Scheduled for presentation during the Regular session "Vision-Language Guided Manipulation" (ThuBT1), Thursday, July 16, 2026, 16:15−16:30, Conference Hall

2026 23rd International Conference on Ubiquitous Robots (UR), July 15-18, 2026, Osaka, Japan

This information is tentative and subject to change. Compiled on October 15, 2026

Abstract

Natural language interfaces can lower the expertise barrier for operating robotic manipulators by allowing users to express goals in everyday speech. This paper presents RANLP-Arm, a modular bilingual voice-to-manipulation system that converts spoken Thai or English instructions into real-time robotic arm actions for tabletop pick-and-place tasks. The system integrates Gemini-based live voice interaction with function calling, stereo 3D object perception and segmentation, persistent object tracking with EMA smoothing and coordinate locking, camera-to-robot calibration with optional IDW residual correction, and TCP/IP robot control in a unified Python architecture for real-time operation. We evaluate the system on a physical robot using 100 programmatically triggered pick-and-place trials and a separate set of 40 bilingual voice-commanded trials. The affine calibration model achieves a mean correspondence error of 3.72 mm on 13 calibration points, while end- to-end experiments achieve 82.0% task completion, 100.0% intent recognition, and 95.0% voice-to-task success. In our trials, all recorded failures were grasp- related; we did not observe failures attributable to ASR, wrong-object grounding, or calibration. These results show that bilingual voice-driven manipulation is practical for tabletop pick-and-place, while the main remaining limitation lies in end-effector robustness.