Evaluation of Large Language Model Responses for 32 Diverse Personality Types Using the Best Worst Method (BWM)

Havaei, Z; Saberi, M; Hussain, OK

Evaluation of Large Language Model Responses for 32 Diverse Personality Types Using the Best Worst Method (BWM)

Havaei, Z Saberi, M

Hussain, OK

Permalink

Publisher:: IEEE
Publication Type:: Conference Proceeding
Citation:: Proceedings - 2024 IEEE International Conference on e-Business Engineering, ICEBE 2024, 2024, 00, pp. 262-271
Issue Date:: 2024-01-01

Closed Access

	Filename	Description	Size
	Evaluation_of_Large_Language_Model_Responses_for_32_Diverse_Personality_Types_Using_the_Best_Worst_Method_BWM.pdf	Published version	613.87 kB	Adobe PDF	View/Open

Copyright Clearance Process

Recently Added
In Progress
Closed Access

This item is closed access and not available.

Full metadata record

Field	Value	Language
dc.contributor.author	Havaei, Z
dc.contributor.author	Saberi, M https://orcid.org/0000-0002-5168-2078
dc.contributor.author	Hussain, OK
dc.date	2024-10-11
dc.date.accessioned	2025-04-02T19:14:01Z
dc.date.available	2025-04-02T19:14:01Z
dc.date.issued	2024-01-01
dc.identifier.citation	Proceedings - 2024 IEEE International Conference on e-Business Engineering, ICEBE 2024, 2024, 00, pp. 262-271
dc.identifier.isbn	979-8-3503-6585-6
dc.identifier.issn	2372-1723
dc.identifier.issn	2472-8527
dc.identifier.uri	http://hdl.handle.net/10453/186518
dc.description.abstract	Today, Chat Generative Pre-trained Transformer (GPT), a generative language model tool created by OpenAI in November 2022, has greatly impacted the AI field. Large language models (LLMs) effectively gather useful information from interaction with humans in diverse fields, including tutoring, translation, customer support, etc. Certainly, ChatGPT is a chatbot designed to be used in conversation to generate human-like responses, and it can answer all kinds of questions in different fields. This study aims to analyze the performance of LLMs by examining how they respond to different prompts based on individual preferences. We applied different personalities and car features to the prompts and employed the Best-Worst Method (BWM). In This approach, ChatGPT generated responses regarding which car features were considered best and worst based on the 32 different personality types. We evaluated the LLM's responses and examined its performance by categorizing all ChatGPT outputs. We categorize them based on each feature chosen and by which type of 32 personality is the best or least feature. This approach enables us to observe how various personalities of humans influence the responses generated by the language models. It will also show us how ChatGPT's decision-making process makes sense to experts in the field of psychology.
dc.language	en
dc.publisher	IEEE
dc.relation.ispartof	Proceedings - 2024 IEEE International Conference on e-Business Engineering, ICEBE 2024
dc.relation.ispartof	IEEE International Conference on e-Business Engineering
dc.relation.isbasedon	10.1109/ICEBE62490.2024.00048
dc.rights	info:eu-repo/semantics/closedAccess
dc.title	Evaluation of Large Language Model Responses for 32 Diverse Personality Types Using the Best Worst Method (BWM)
dc.type	Conference Proceeding
utslib.citation.volume	00
utslib.location.activity	Shanghai, China
pubs.organisational-group	University of Technology Sydney
pubs.organisational-group	University of Technology Sydney/Faculty of Engineering and Information Technology
pubs.organisational-group	University of Technology Sydney/Faculty of Engineering and Information Technology/School of Computer Science
pubs.organisational-group	University of Technology Sydney/UTS Groups
pubs.organisational-group	University of Technology Sydney/UTS Groups/Data Science Institute (DSI)
utslib.copyright.status	closed_access	*
pubs.consider-herdc	false
dc.date.updated	2025-04-02T19:14:00Z
pubs.finish-date	2024-10-13
pubs.place-of-publication	Piscataway, USA
pubs.publication-status	Published
pubs.start-date	2024-10-11
pubs.volume	00
dc.location	Piscataway, USA

Abstract:

Today, Chat Generative Pre-trained Transformer (GPT), a generative language model tool created by OpenAI in November 2022, has greatly impacted the AI field. Large language models (LLMs) effectively gather useful information from interaction with humans in diverse fields, including tutoring, translation, customer support, etc. Certainly, ChatGPT is a chatbot designed to be used in conversation to generate human-like responses, and it can answer all kinds of questions in different fields. This study aims to analyze the performance of LLMs by examining how they respond to different prompts based on individual preferences. We applied different personalities and car features to the prompts and employed the Best-Worst Method (BWM). In This approach, ChatGPT generated responses regarding which car features were considered best and worst based on the 32 different personality types. We evaluated the LLM's responses and examined its performance by categorizing all ChatGPT outputs. We categorize them based on each feature chosen and by which type of 32 personality is the best or least feature. This approach enables us to observe how various personalities of humans influence the responses generated by the language models. It will also show us how ChatGPT's decision-making process makes sense to experts in the field of psychology.

Please use this identifier to cite or link to this item:

http://hdl.handle.net/10453/186518