Understanding and Measuring User Experience in Conversational Interfaces

Kocaballi, AB; Laranjo, L; Coiera, E

Understanding and Measuring User Experience in Conversational Interfaces

Kocaballi, AB Laranjo, L Coiera, E

Permalink

Publisher:: OXFORD UNIV PRESS
Publication Type:: Journal Article
Citation:: Interacting with Computers, 2019, 31, (2), pp. 192-207
Issue Date:: 2019-03-01

Closed Access

	Filename	Description	Size
	ILL Article to Scott M_UNDERSTANDING AND MEASURING USER EXPERIENCE IN CONVERSATIONAL INTERFACES. 19.5.20.pdf	Published version	347.35 kB	Adobe PDF	View/Open

Copyright Clearance Process

Recently Added
In Progress
Closed Access

This item is closed access and not available.

Full metadata record

Field	Value	Language
dc.contributor.author	Kocaballi, AB
dc.contributor.author	Laranjo, L
dc.contributor.author	Coiera, E
dc.date.accessioned	2020-05-19T00:38:45Z
dc.date.available	2020-05-19T00:38:45Z
dc.date.issued	2019-03-01
dc.identifier.citation	Interacting with Computers, 2019, 31, (2), pp. 192-207
dc.identifier.issn	0953-5438
dc.identifier.issn	1873-7951
dc.identifier.uri	http://hdl.handle.net/10453/140802
dc.description.abstract	© 2019 The Author(s) 2019. Published by Oxford University Press on behalf of The British Computer Society. All rights reserved. For Permissions, please email: journals.permissions@oup.com. Although various methods have been developed to evaluate conversational interfaces, there has been a lack of methods specifically focusing on evaluating user experience. This paper reviews the understandings of user experience (UX) in conversational interfaces literature and examines the six questionnaires commonly used for evaluating conversational systems in order to assess the potential suitability of these questionnaires to measure different UX dimensions in that context. The method to examine the questionnaires involved developing an assessment framework for main UX dimensions with relevant attributes and coding the items in the questionnaires according to the framework. The results show that (i) the understandings of UX notably differed in literature; (ii) four questionnaires included assessment items, in varying extents, to measure hedonic, aesthetic and pragmatic dimensions of UX; (iii) while the dimension of affect was covered by two questionnaires, playfulness, motivation, and frustration dimensions were covered by one questionnaire only. The largest coverage of UX dimensions has been provided by the Subjective Assessment of Speech System Interfaces (SASSI). We recommend using multiple questionnaires to obtain a more complete measurement of user experience or improve the assessment of a particular UX dimension. RESEARCH HIGHLIGHTS: Varying understandings of UX in conversational interfaces literature. A UX assessment framework with UX dimensions and their relevant attributes. Descriptions of the six main questionnaires for evaluating conversational interfaces. A comparison of the six questionnaires based on their coverage of UX dimensions.
dc.language	English
dc.publisher	OXFORD UNIV PRESS
dc.relation.ispartof	Interacting with Computers
dc.relation.isbasedon	10.1093/iwc/iwz015
dc.rights	info:eu-repo/semantics/closedAccess
dc.subject	0806 Information Systems, 1203 Design Practice and Management, 1702 Cognitive Sciences
dc.subject.classification	Human Factors
dc.title	Understanding and Measuring User Experience in Conversational Interfaces
dc.type	Journal Article
utslib.citation.volume	31
utslib.for	0806 Information Systems
utslib.for	1203 Design Practice and Management
utslib.for	1702 Cognitive Sciences
pubs.organisational-group	/University of Technology Sydney/Faculty of Engineering and Information Technology
pubs.organisational-group	/University of Technology Sydney/Faculty of Engineering and Information Technology/School of Computer Science
pubs.organisational-group	/University of Technology Sydney
utslib.copyright.status	closed_access	*
dc.date.updated	2020-05-19T00:38:41Z
pubs.issue	2
pubs.publication-status	Published
pubs.volume	31
utslib.start-page	192
utslib.citation.issue	2

Abstract:

© 2019 The Author(s) 2019. Published by Oxford University Press on behalf of The British Computer Society. All rights reserved. For Permissions, please email: journals.permissions@oup.com. Although various methods have been developed to evaluate conversational interfaces, there has been a lack of methods specifically focusing on evaluating user experience. This paper reviews the understandings of user experience (UX) in conversational interfaces literature and examines the six questionnaires commonly used for evaluating conversational systems in order to assess the potential suitability of these questionnaires to measure different UX dimensions in that context. The method to examine the questionnaires involved developing an assessment framework for main UX dimensions with relevant attributes and coding the items in the questionnaires according to the framework. The results show that (i) the understandings of UX notably differed in literature; (ii) four questionnaires included assessment items, in varying extents, to measure hedonic, aesthetic and pragmatic dimensions of UX; (iii) while the dimension of affect was covered by two questionnaires, playfulness, motivation, and frustration dimensions were covered by one questionnaire only. The largest coverage of UX dimensions has been provided by the Subjective Assessment of Speech System Interfaces (SASSI). We recommend using multiple questionnaires to obtain a more complete measurement of user experience or improve the assessment of a particular UX dimension. RESEARCH HIGHLIGHTS: Varying understandings of UX in conversational interfaces literature. A UX assessment framework with UX dimensions and their relevant attributes. Descriptions of the six main questionnaires for evaluating conversational interfaces. A comparison of the six questionnaires based on their coverage of UX dimensions.

Please use this identifier to cite or link to this item:

http://hdl.handle.net/10453/140802