Design, Integration, and Field Testing of a Digital Twin-Based Teleoperated Rock Scaling Robot

Le, DT; Sutjipto, S; Nguyen, K; Paul, G

Design, Integration, and Field Testing of a Digital Twin-Based Teleoperated Rock Scaling Robot

Le, DT Sutjipto, S

Nguyen, K

Paul, G

Permalink

Publisher:: IEEE Robotics & Automation Society (RAS)
Publication Type:: Journal Article
Citation:: IEEE Transactions on Field Robotics, 2025, 2, pp. 188-207
Issue Date:: 2025-03-10

Open Access

Copyright Clearance Process

Recently Added
In Progress
Open Access

This item is open access.

Adobe PDF

Download Published versionAdobe PDF (16.1 MB)

View on publisher's site

View statistics

Full metadata record

Field	Value	Language
dc.contributor.author	Le, DT
dc.contributor.author	Sutjipto, S https://orcid.org/0000-0002-0697-0265
dc.contributor.author	Nguyen, K https://orcid.org/0000-0003-3233-4972
dc.contributor.author	Paul, G https://orcid.org/0000-0002-3478-0020
dc.date.accessioned	2026-01-12T05:34:05Z
dc.date.available	2024-12-01
dc.date.available	2026-01-12T05:34:05Z
dc.date.issued	2025-03-10
dc.identifier.citation	IEEE Transactions on Field Robotics, 2025, 2, pp. 188-207
dc.identifier.issn	2997-1101
dc.identifier.issn	2997-1101
dc.identifier.uri	http://hdl.handle.net/10453/191672
dc.description.abstract	This article presents the design, integration, and field testing of a digital twin-based teleoperated rock scaling robot aimed at improving safety in mining operations. Traditional rock scaling, which involves the removal of loose rocks to prevent rockfall, poses significant risks to mine site workers. The proposed solution is a teleoperated custom mobile manipulator capable of rope-based abseiling locomotion, equipped with an air chipper end-effector. Teleoperation is facilitated by live digital twins of the robot and environment, with a virtual reality (VR) interface that allows operators to perform rock scaling tasks within an immersive virtual reconstruction of the remote scene. The robot’s hardware design and sensing capabilities are detailed, along with the system’s teleoperation architecture. Key components include the integration of an optimized, hardware-accelerated, image-based point cloud streaming implementation; a markerless depth-camera extrinsic calibration process suitable for field settings; and the system’s teleoperation interfaces featuring a cyber-physical VR interface with affordance feedback. Field tests at a sandstone quarry and an open-pit mine demonstrate significant improvements in operator safety and highlight the system’s ability to withstand harsh mining environments while performing teleoperated rock scaling at its current scaled-down size and power. We collected and analyzed user data from rope access technicians with no prior experience in robot teleoperation or VR. The results suggest the system’s intuitiveness with learning effects over time. Lessons from these site trials, including hardware and software limitations, are discussed, providing directions for further robot design improvements and enhancements to the digital twin teleoperation architecture.
dc.language	en
dc.publisher	IEEE Robotics & Automation Society (RAS)
dc.relation	Australian Mathematical Sciences Institute
dc.relation	http://purl.org/au-research/grants/arc/IC200100001
dc.relation.ispartof	IEEE Transactions on Field Robotics
dc.relation.isbasedon	10.1109/TFR.2025.3547807
dc.rights	info:eu-repo/semantics/openAccess
dc.title	Design, Integration, and Field Testing of a Digital Twin-Based Teleoperated Rock Scaling Robot
dc.type	Journal Article
utslib.citation.volume	2
pubs.organisational-group	University of Technology Sydney
pubs.organisational-group	University of Technology Sydney/Faculty of Engineering and Information Technology
pubs.organisational-group	University of Technology Sydney/Faculty of Engineering and Information Technology/School of Mechanical and Mechatronic Engineering
pubs.organisational-group	University of Technology Sydney/UTS Groups
pubs.organisational-group	University of Technology Sydney/UTS Groups/Robotics Institute (RI)
pubs.organisational-group	University of Technology Sydney/UTS Groups/Robotics Institute (RI)/Robotics Institute (RI) Associate Members
pubs.organisational-group	University of Technology Sydney/Faculty of Engineering and Information Technology/Engineering and IT Related HDR Students
utslib.copyright.status	open_access	*
pubs.consider-herdc	true
dc.rights.license	This work is licensed under a Creative Commons Attribution 4.0 International License (CC BY 4.0). To view a copy of this license, visit https://creativecommons.org/licenses/by/4.0/
dc.date.updated	2026-01-12T05:34:03Z
pubs.publication-status	Published online
pubs.volume	2

Abstract:

This article presents the design, integration, and field testing of a digital twin-based teleoperated rock scaling robot aimed at improving safety in mining operations. Traditional rock scaling, which involves the removal of loose rocks to prevent rockfall, poses significant risks to mine site workers. The proposed solution is a teleoperated custom mobile manipulator capable of rope-based abseiling locomotion, equipped with an air chipper end-effector. Teleoperation is facilitated by live digital twins of the robot and environment, with a virtual reality (VR) interface that allows operators to perform rock scaling tasks within an immersive virtual reconstruction of the remote scene. The robot’s hardware design and sensing capabilities are detailed, along with the system’s teleoperation architecture. Key components include the integration of an optimized, hardware-accelerated, image-based point cloud streaming implementation; a markerless depth-camera extrinsic calibration process suitable for field settings; and the system’s teleoperation interfaces featuring a cyber-physical VR interface with affordance feedback. Field tests at a sandstone quarry and an open-pit mine demonstrate significant improvements in operator safety and highlight the system’s ability to withstand harsh mining environments while performing teleoperated rock scaling at its current scaled-down size and power. We collected and analyzed user data from rope access technicians with no prior experience in robot teleoperation or VR. The results suggest the system’s intuitiveness with learning effects over time. Lessons from these site trials, including hardware and software limitations, are discussed, providing directions for further robot design improvements and enhancements to the digital twin teleoperation architecture.

Please use this identifier to cite or link to this item:

http://hdl.handle.net/10453/191672