End-to-end Multi-Instance Robotic Reaching from Monocular Vision

Zhuang, Z; Yu, X; Mahony, R

End-to-end Multi-Instance Robotic Reaching from Monocular Vision

Zhuang, Z Yu, X

Mahony, R

Permalink

Publisher:: IEEE
Publication Type:: Conference Proceeding
Citation:: 2021 IEEE International Conference on Robotics and Automation (ICRA), 2021, 00, pp. 12974-12980
Issue Date:: 2021-10-18

Closed Access

	Filename	Description	Size
	End-to-end_Multi-Instance_Robotic_Reaching_from_Monocular_Vision.pdf	Published version	14.08 MB	Adobe PDF	View/Open

Copyright Clearance Process

Recently Added
In Progress
Closed Access

This item is closed access and not available.

Full metadata record

Field	Value	Language
dc.contributor.author	Zhuang, Z
dc.contributor.author	Yu, X https://orcid.org/0000-0002-0269-5649
dc.contributor.author	Mahony, R
dc.date	2021-05-30
dc.date.accessioned	2022-04-11T23:18:48Z
dc.date.available	2022-04-11T23:18:48Z
dc.date.issued	2021-10-18
dc.identifier.citation	2021 IEEE International Conference on Robotics and Automation (ICRA), 2021, 00, pp. 12974-12980
dc.identifier.uri	http://hdl.handle.net/10453/156093
dc.description.abstract	Multi-instance scenes are especially challenging for end-to-end visuomotor (image-to-control) learning algorithms. "Pipeline" visual servo control algorithms use separate detection, selection and servo stages, allowing algorithms to focus on a single object instance during servo control. End-to-end systems do not have separate detection and selection stages and need to address the visual ambiguities introduced by the presence of an arbitrary number of visually identical or similar objects during servo control. However, end-to-end schemes avoid embedding errors from detection and selection stages in the servo control behaviour, are more dynamically robust to changing scenes and are algorithmically simpler. In this paper, we present a reactive real-time end-to-end visuomotor learning algorithm for multi-instance reaching. The proposed algorithm uses a monocular RGB image and the manipulator’s joint angles as the input to a light-weight fully-convolutional network (FCN) to generate control candidates. A key innovation of the proposed method is identifying the optimal control candidate by regressing a control-Lyapunov function (cLf) value. The multi-instance capability emerges naturally from the stability analysis associated with the cLf formulation. We demonstrate the proposed algorithm effectively reaching and grasping objects from different categories on a table-top amid other instances and distractors from an over-the-shoulder monocular RGB camera. The network is able to run up to ∼160 fps during inference on one GTX 1080 Ti GPU.
dc.language	en
dc.publisher	IEEE
dc.relation.ispartof	2021 IEEE International Conference on Robotics and Automation (ICRA)
dc.relation.ispartof	2021 IEEE International Conference on Robotics and Automation
dc.relation.isbasedon	10.1109/icra48506.2021.9561518
dc.rights	info:eu-repo/semantics/closedAccess
dc.title	End-to-end Multi-Instance Robotic Reaching from Monocular Vision
dc.type	Conference Proceeding
utslib.citation.volume	00
utslib.location.activity	Xi'an, China
pubs.organisational-group	/University of Technology Sydney
pubs.organisational-group	/University of Technology Sydney/Faculty of Engineering and Information Technology
pubs.organisational-group	/University of Technology Sydney/Strength - AAII - Australian Artificial Intelligence Institute
pubs.organisational-group	/University of Technology Sydney/Faculty of Engineering and Information Technology/School of Computer Science
utslib.copyright.status	closed_access	*
dc.date.updated	2022-04-11T23:18:46Z
pubs.finish-date	2021-06-05
pubs.publication-status	Published
pubs.start-date	2021-05-30
pubs.volume	00

Abstract:

Multi-instance scenes are especially challenging for end-to-end visuomotor (image-to-control) learning algorithms. "Pipeline" visual servo control algorithms use separate detection, selection and servo stages, allowing algorithms to focus on a single object instance during servo control. End-to-end systems do not have separate detection and selection stages and need to address the visual ambiguities introduced by the presence of an arbitrary number of visually identical or similar objects during servo control. However, end-to-end schemes avoid embedding errors from detection and selection stages in the servo control behaviour, are more dynamically robust to changing scenes and are algorithmically simpler. In this paper, we present a reactive real-time end-to-end visuomotor learning algorithm for multi-instance reaching. The proposed algorithm uses a monocular RGB image and the manipulator’s joint angles as the input to a light-weight fully-convolutional network (FCN) to generate control candidates. A key innovation of the proposed method is identifying the optimal control candidate by regressing a control-Lyapunov function (cLf) value. The multi-instance capability emerges naturally from the stability analysis associated with the cLf formulation. We demonstrate the proposed algorithm effectively reaching and grasping objects from different categories on a table-top amid other instances and distractors from an over-the-shoulder monocular RGB camera. The network is able to run up to ∼160 fps during inference on one GTX 1080 Ti GPU.

Please use this identifier to cite or link to this item:

http://hdl.handle.net/10453/156093