Supervised ensemble sentiment-based framework to measure chatbot quality of services

Almansor, EH; Hussain, FK; Hussain, OK

Supervised ensemble sentiment-based framework to measure chatbot quality of services

Almansor, EH Hussain, FK Hussain, OK

Permalink

Publisher:: Springer
Publication Type:: Journal Article
Citation:: Computing, 2021, 103, (3), pp. 491-507
Issue Date:: 2021-01-01

Closed Access

	Filename	Description	Size
	Supervised_ensemble_sentiment-.pdf		782.56 kB	Adobe PDF	View/Open

Copyright Clearance Process

Recently Added
In Progress
Closed Access

This item is closed access and not available.

Full metadata record

Field	Value	Language
dc.contributor.author	Almansor, EH
dc.contributor.author	Hussain, FK
dc.contributor.author	Hussain, OK
dc.date.accessioned	2022-05-19T06:02:18Z
dc.date.available	2022-05-19T06:02:18Z
dc.date.issued	2021-01-01
dc.identifier.citation	Computing, 2021, 103, (3), pp. 491-507
dc.identifier.issn	0010-485X
dc.identifier.issn	1436-5057
dc.identifier.uri	http://hdl.handle.net/10453/157533
dc.description.abstract	Developing an intelligent chatbot has evolved in the last few years to become a trending topic in the area of computer science. However, a chatbot often fails to understand the user’s intent, which can lead to the generation of inappropriate responses that cause dialogue breakdown and user dissatisfaction. Detecting the dialogue breakdown is essential to improve the performance of the chatbot and increase user satisfaction. Recent approaches have focused on modeling conversation breakdown using serveral approaches, including supervised and unsupervised approaches. Unsupervised approach relay heavy datasets, which make it challenging to apply it to the breakdown task. Another challenge facing predicting breakdown in conversation is the bias of human annotation for the dataset and the handling process for the breakdown. To tackle this challenge, we have developed a supervised ensemble automated approach that measures Chatbot Quality of Service (CQoS) based on dialogue breakdown. The proposed approach is able to label the datasets based on sentiment considering the context of the conversion to predict the breakdown. In this paper we aim to detect the affect of sentiment change of each speaker in a conversation. Furthermore, we use the supervised ensemble model to measure the CQoS based on breakdown. Then we handle this problem by using a hand-over mechanism that transfers the user to a live agent. Based on this idea, we perform several experiments across several datasets and state-of-the-art models, and we find that using sentiment as a trigger for breakdown outperforms human annotation. Overall, we infer that knowledge acquired from the supervised ensemble model can indeed help to measure CQoS based on detecting the breakdown in conversation.
dc.language	en
dc.publisher	Springer
dc.relation.ispartof	Computing
dc.relation.isbasedon	10.1007/s00607-020-00863-0
dc.rights	info:eu-repo/semantics/closedAccess
dc.subject	01 Mathematical Sciences, 08 Information and Computing Sciences
dc.subject.classification	Numerical & Computational Mathematics
dc.title	Supervised ensemble sentiment-based framework to measure chatbot quality of services
dc.type	Journal Article
utslib.citation.volume	103
utslib.for	01 Mathematical Sciences
utslib.for	08 Information and Computing Sciences
pubs.organisational-group	/University of Technology Sydney
pubs.organisational-group	/University of Technology Sydney/Faculty of Engineering and Information Technology
pubs.organisational-group	/University of Technology Sydney/Strength - AAII - Australian Artificial Intelligence Institute
pubs.organisational-group	/University of Technology Sydney/Faculty of Engineering and Information Technology/School of Computer Science
utslib.copyright.status	closed_access	*
pubs.consider-herdc	false
dc.date.updated	2022-05-19T06:02:16Z
pubs.issue	3
pubs.publication-status	Published
pubs.volume	103
utslib.citation.issue	3

Abstract:

Developing an intelligent chatbot has evolved in the last few years to become a trending topic in the area of computer science. However, a chatbot often fails to understand the user’s intent, which can lead to the generation of inappropriate responses that cause dialogue breakdown and user dissatisfaction. Detecting the dialogue breakdown is essential to improve the performance of the chatbot and increase user satisfaction. Recent approaches have focused on modeling conversation breakdown using serveral approaches, including supervised and unsupervised approaches. Unsupervised approach relay heavy datasets, which make it challenging to apply it to the breakdown task. Another challenge facing predicting breakdown in conversation is the bias of human annotation for the dataset and the handling process for the breakdown. To tackle this challenge, we have developed a supervised ensemble automated approach that measures Chatbot Quality of Service (CQoS) based on dialogue breakdown. The proposed approach is able to label the datasets based on sentiment considering the context of the conversion to predict the breakdown. In this paper we aim to detect the affect of sentiment change of each speaker in a conversation. Furthermore, we use the supervised ensemble model to measure the CQoS based on breakdown. Then we handle this problem by using a hand-over mechanism that transfers the user to a live agent. Based on this idea, we perform several experiments across several datasets and state-of-the-art models, and we find that using sentiment as a trigger for breakdown outperforms human annotation. Overall, we infer that knowledge acquired from the supervised ensemble model can indeed help to measure CQoS based on detecting the breakdown in conversation.

Please use this identifier to cite or link to this item:

http://hdl.handle.net/10453/157533