Augmented 3D hands: a gesture-based mixed reality system for distributed collaboration

Huang, W; Alem, L; Tecchia, F; Duh, HBL

Augmented 3D hands: a gesture-based mixed reality system for distributed collaboration

Huang, W

Alem, L

Tecchia, F Duh, HBL

Permalink

Publisher:: SPRINGER
Publication Type:: Journal Article
Citation:: Journal on Multimodal User Interfaces, 2018, 12, (2), pp. 77-89
Issue Date:: 2018-06-01

Closed Access

	Filename	Description	Size
	s12193-017-0250-2.pdf	Published version	1.96 MB	Adobe PDF	View/Open

Copyright Clearance Process

Recently Added
In Progress
Closed Access

This item is closed access and not available.

Full metadata record

Field	Value	Language
dc.contributor.author	Huang, W https://orcid.org/0000-0002-5190-7839
dc.contributor.author	Alem, L https://orcid.org/0000-0002-7802-3590
dc.contributor.author	Tecchia, F
dc.contributor.author	Duh, HBL
dc.date.accessioned	2022-08-24T04:28:02Z
dc.date.available	2022-08-24T04:28:02Z
dc.date.issued	2018-06-01
dc.identifier.citation	Journal on Multimodal User Interfaces, 2018, 12, (2), pp. 77-89
dc.identifier.issn	1783-7677
dc.identifier.issn	1783-8738
dc.identifier.uri	http://hdl.handle.net/10453/160782
dc.description.abstract	Distributed collaborations between two or more participants on a task involving tangible artifacts (e.g., a machine, a patient, a tool) have become increasingly common in recent years due to rapid development in information and communication technologies. In this paper we focus on a specific type of remote-collaboration system where a remote helper guides a local worker using audio communication and hand gestures to perform a repair or a maintenance task. An established ICT approach to supporting this type of collaboration is to provide a shared visual space and some forms of remote gesture. The shared space typically consists of a video capture of the remote workspace which is then displayed on a 2D screen. However, this type of approach has its limitations. Firstly, it does not provide the helper with sufficient understanding of the spatial relationships between objects in the remote workspace. Secondly, it does not allow the helper to gesture in 3D. In an attempt to address these issues, we propose a Mixed Reality multimodal system that improves on previous 2D systems by introducing 3D real-time capturing and rendering of both the remote workspace and the helping hands and by creating a 3D shared visual space as a result of co-locating the remote workspace with the helping hands. In this system, we explore the possibility of increasing the feeling of immersion and co-presence by using head tracking, stereoscopic rendering, inter-occlusion handling and virtual shadowing. In this paper, we introduce HandsIn3D, a system that has been developed for the purpose of the proof of concept. We also present the results of experiments to verify the feasibility of our approach.
dc.language	English
dc.publisher	SPRINGER
dc.relation.ispartof	Journal on Multimodal User Interfaces
dc.relation.isbasedon	10.1007/s12193-017-0250-2
dc.rights	info:eu-repo/semantics/closedAccess
dc.subject	0801 Artificial Intelligence and Image Processing, 0806 Information Systems, 1203 Design Practice and Management
dc.title	Augmented 3D hands: a gesture-based mixed reality system for distributed collaboration
dc.type	Journal Article
utslib.citation.volume	12
utslib.for	0801 Artificial Intelligence and Image Processing
utslib.for	0806 Information Systems
utslib.for	1203 Design Practice and Management
pubs.organisational-group	/University of Technology Sydney
pubs.organisational-group	/University of Technology Sydney/Provost
pubs.organisational-group	/University of Technology Sydney/Faculty of Transdisciplinary Innovation
pubs.organisational-group	/University of Technology Sydney/Provost/TD School
utslib.copyright.status	closed_access	*
dc.date.updated	2022-08-24T04:28:01Z
pubs.issue	2
pubs.publication-status	Published
pubs.volume	12
utslib.citation.issue	2

Abstract:

Distributed collaborations between two or more participants on a task involving tangible artifacts (e.g., a machine, a patient, a tool) have become increasingly common in recent years due to rapid development in information and communication technologies. In this paper we focus on a specific type of remote-collaboration system where a remote helper guides a local worker using audio communication and hand gestures to perform a repair or a maintenance task. An established ICT approach to supporting this type of collaboration is to provide a shared visual space and some forms of remote gesture. The shared space typically consists of a video capture of the remote workspace which is then displayed on a 2D screen. However, this type of approach has its limitations. Firstly, it does not provide the helper with sufficient understanding of the spatial relationships between objects in the remote workspace. Secondly, it does not allow the helper to gesture in 3D. In an attempt to address these issues, we propose a Mixed Reality multimodal system that improves on previous 2D systems by introducing 3D real-time capturing and rendering of both the remote workspace and the helping hands and by creating a 3D shared visual space as a result of co-locating the remote workspace with the helping hands. In this system, we explore the possibility of increasing the feeling of immersion and co-presence by using head tracking, stereoscopic rendering, inter-occlusion handling and virtual shadowing. In this paper, we introduce HandsIn3D, a system that has been developed for the purpose of the proof of concept. We also present the results of experiments to verify the feasibility of our approach.

Please use this identifier to cite or link to this item:

http://hdl.handle.net/10453/160782