Comparison of Rating Scale and Pairwise Comparison Methods for Measuring Human Co-worker Subjective Impression of Robot during Physical Human-Robot Collaboration

Wang, Q; Wang, Z; Carmichael, MG; Liu, D; Lin, C-T

Comparison of Rating Scale and Pairwise Comparison Methods for Measuring Human Co-worker Subjective Impression of Robot during Physical Human-Robot Collaboration

Wang, Q Wang, Z

Carmichael, MG Liu, D

Lin, C-T

Permalink

Publisher:: IEEE
Publication Type:: Conference Proceeding
Citation:: 2024 IEEE International Conference on Robotics and Automation (ICRA), 2024, 00, pp. 907-913
Issue Date:: 2024-01-01

Embargoed

	Filename	Description	Size
	Comparison of Rating Scale and Pairwise Comparison Methods for Measuring Human Co-worker Subjective Impression of Robot during Physical Human-Robot Collaboration.pdf	Accepted version	609.73 kB	Adobe PDF	View/Open

Copyright Clearance Process

Recently Added
In Progress
Embargoed
Open Access

This item is currently unavailable due to the publisher's embargo.

The embargo period expires on 8 Aug 2026

Full metadata record

Field	Value	Language
dc.contributor.author	Wang, Q
dc.contributor.author	Wang, Z https://orcid.org/0000-0001-7167-123X
dc.contributor.author	Carmichael, MG
dc.contributor.author	Liu, D https://orcid.org/0000-0002-1581-5582
dc.contributor.author	Lin, C-T
dc.date	2024-05-13
dc.date.accessioned	2024-10-09T03:38:06Z
dc.date.available	2024-10-09T03:38:06Z
dc.date.issued	2024-01-01
dc.identifier.citation	2024 IEEE International Conference on Robotics and Automation (ICRA), 2024, 00, pp. 907-913
dc.identifier.isbn	979-8-3503-8458-1
dc.identifier.issn	1050-4729
dc.identifier.uri	http://hdl.handle.net/10453/181317
dc.description.abstract	The Rating Scale method has been long deemed the standard for measuring subjective perceptions However in the field of physical human robot collaboration pHRC its aptness should be put under scrutiny due to inherent challenges such as response bias between subject variations and the granularity nature Individual variances can introduce significant bias in the rating scale results A high granularity in the scale could overwhelm participants leading to unclear and biased responses while a low granularity may gloss over the fine nuances of human feelings Additionally there s a notable risk of receiving careless responses which compromise data reliability Recognizing these challenges this paper proposes the application of Pairwise Comparison PC in pHRC an alternative survey technique that emphasizes direct comparisons between items on the defined criteria By using the NASA Task Load Index NASA TLX as a template RS and PC questionnaires are designed and used in a series of pHRC experiments Our preliminary findings suggest that PC is more precise and robust than the rating scale method Compared to RS PC fosters authentic participant interests in the experiment by intuitive question design and reducing the experimental duration Besides the accuracy and reliability of PC are also found to be consistent regardless of the variations in our experimental procedure design
dc.language	en
dc.publisher	IEEE
dc.relation	http://purl.org/au-research/grants/arc/DP210101093
dc.relation.ispartof	2024 IEEE International Conference on Robotics and Automation (ICRA)
dc.relation.ispartof	2024 IEEE International Conference on Robotics and Automation
dc.relation.isbasedon	10.1109/icra57147.2024.10611050
dc.rights	info:eu-repo/semantics/embargoedAccess
dc.title	Comparison of Rating Scale and Pairwise Comparison Methods for Measuring Human Co-worker Subjective Impression of Robot during Physical Human-Robot Collaboration
dc.type	Conference Proceeding
utslib.citation.volume	00
utslib.location.activity	Yokohama, Japan
pubs.organisational-group	University of Technology Sydney
pubs.organisational-group	University of Technology Sydney/Faculty of Engineering and Information Technology
pubs.organisational-group	University of Technology Sydney/Faculty of Engineering and Information Technology/School of Mechanical and Mechatronic Engineering
pubs.organisational-group	University of Technology Sydney/Faculty of Engineering and Information Technology/School of Computer Science
pubs.organisational-group	University of Technology Sydney/UTS Groups
pubs.organisational-group	University of Technology Sydney/UTS Groups/Australian Artificial Intelligence Institute (AAII)
pubs.organisational-group	University of Technology Sydney/UTS Groups/Robotics Institute (RI)
pubs.organisational-group	University of Technology Sydney/UTS Groups/Centre for Advanced Manufacturing (CAM)
pubs.organisational-group	University of Technology Sydney/UTS Groups/Centre for Advanced Manufacturing (CAM)/Centre for Advanced Manufacturing (CAM) Associate Member
utslib.copyright.status	embargoed	*
utslib.copyright.embargo	2026-08-08T00:00:00+1000Z
dc.date.updated	2024-10-09T03:38:04Z
pubs.finish-date	2024-05-17
pubs.place-of-publication	Piscataway, USA
pubs.publication-status	Published
pubs.start-date	2024-05-13
pubs.volume	00
dc.location	Piscataway, USA

Abstract:

The Rating Scale method has been long deemed the standard for measuring subjective perceptions However in the field of physical human robot collaboration pHRC its aptness should be put under scrutiny due to inherent challenges such as response bias between subject variations and the granularity nature Individual variances can introduce significant bias in the rating scale results A high granularity in the scale could overwhelm participants leading to unclear and biased responses while a low granularity may gloss over the fine nuances of human feelings Additionally there s a notable risk of receiving careless responses which compromise data reliability Recognizing these challenges this paper proposes the application of Pairwise Comparison PC in pHRC an alternative survey technique that emphasizes direct comparisons between items on the defined criteria By using the NASA Task Load Index NASA TLX as a template RS and PC questionnaires are designed and used in a series of pHRC experiments Our preliminary findings suggest that PC is more precise and robust than the rating scale method Compared to RS PC fosters authentic participant interests in the experiment by intuitive question design and reducing the experimental duration Besides the accuracy and reliability of PC are also found to be consistent regardless of the variations in our experimental procedure design

Please use this identifier to cite or link to this item:

http://hdl.handle.net/10453/181317