Benchmarking large language models for supply chain risk identification: an extended evaluation within the LARD-SC framework

Zhao, M; Hussain, O; Zhang, Y; Saberi, M; Leshob, A

Benchmarking large language models for supply chain risk identification: an extended evaluation within the LARD-SC framework

Zhao, M Hussain, O Zhang, Y Saberi, M

Leshob, A

Permalink

Publisher:: SPRINGERNATURE
Publication Type:: Journal Article
Citation:: Service Oriented Computing and Applications, 2025
Issue Date:: 2025-01-01

Open Access

Copyright Clearance Process

Recently Added
In Progress
Open Access

This item is open access.

Adobe PDF

Download Published versionAdobe PDF (2.03 MB)

View on publisher's site

View statistics

Full metadata record

Field	Value	Language
dc.contributor.author	Zhao, M
dc.contributor.author	Hussain, O
dc.contributor.author	Zhang, Y
dc.contributor.author	Saberi, M https://orcid.org/0000-0002-5168-2078
dc.contributor.author	Leshob, A
dc.date.accessioned	2026-01-29T23:45:46Z
dc.date.available	2026-01-29T23:45:46Z
dc.date.issued	2025-01-01
dc.identifier.citation	Service Oriented Computing and Applications, 2025
dc.identifier.issn	1863-2386
dc.identifier.issn	1863-2394
dc.identifier.uri	http://hdl.handle.net/10453/192545
dc.description.abstract	Operational resilience in modern global supply chains depends on timely and accurate identification of emerging risks. While daily news has become a primary source for such insights, the sheer volume and unstructured nature of these data pose significant analytical challenges, requiring advanced tools to extract relevant and actionable information. This paper introduces an extended evaluation of the LARD-SC framework, a service-oriented architecture for supply chain risk management, by benchmarking five diverse variants of the large language model (LLM) in their capacity to detect, classify, and interpret risks. Drawing on a curated set of 120 real-world news articles on Apple’s Tier 1 suppliers, we adopt a standardized, prompt-based assessment to compare GPT-3.5 turbo, GPT-4o, GPT-4o mini, Claude 3.5 Sonnet, and Claude 3.5 Haiku. Using expert-reviewed metrics, namely the Risk Validation Rate (RVR), Potential Risk Rate (PRR), and False Identification Rate (FIR), we derive a comprehensive Relative Performance Index (RPI) for comparison. Our analysis confirms that advanced GPT-4o variants produce the most consistent accurate risk identifications, achieving higher proportions of validated outcomes while minimizing false positives. Through these results, we highlight the significant promise of LLM-driven analytics for early risk detection in complex supply chains, along with practical considerations such as the influence of prompt engineering, interpretability demands, and the impact of data availability. The findings offer a blueprint for organizations seeking to improve resilience by systematically harnessing the capabilities of LLM within service-oriented risk management ecosystems.
dc.language	English
dc.publisher	SPRINGERNATURE
dc.relation.ispartof	Service Oriented Computing and Applications
dc.relation.isbasedon	10.1007/s11761-025-00474-7
dc.rights	info:eu-repo/semantics/openAccess
dc.subject	0805 Distributed Computing, 0806 Information Systems
dc.subject.classification	Information Systems
dc.subject.classification	4606 Distributed computing and systems software
dc.title	Benchmarking large language models for supply chain risk identification: an extended evaluation within the LARD-SC framework
dc.type	Journal Article
utslib.for	0805 Distributed Computing
utslib.for	0806 Information Systems
pubs.organisational-group	University of Technology Sydney
pubs.organisational-group	University of Technology Sydney/Faculty of Engineering and Information Technology
pubs.organisational-group	University of Technology Sydney/Faculty of Engineering and Information Technology/School of Computer Science
pubs.organisational-group	University of Technology Sydney/UTS Groups
pubs.organisational-group	University of Technology Sydney/UTS Groups/Data Science Institute (DSI)
pubs.organisational-group	University of Technology Sydney/UTS Groups/The Trustworthy Digital Society
utslib.copyright.status	open_access	*
dc.rights.license	This work is licensed under a Creative Commons Attribution 4.0 International License (CC BY 4.0). To view a copy of this license, visit https://creativecommons.org/licenses/by/4.0/
dc.date.updated	2026-01-29T23:45:45Z
pubs.publication-status	Published

Abstract:

Operational resilience in modern global supply chains depends on timely and accurate identification of emerging risks. While daily news has become a primary source for such insights, the sheer volume and unstructured nature of these data pose significant analytical challenges, requiring advanced tools to extract relevant and actionable information. This paper introduces an extended evaluation of the LARD-SC framework, a service-oriented architecture for supply chain risk management, by benchmarking five diverse variants of the large language model (LLM) in their capacity to detect, classify, and interpret risks. Drawing on a curated set of 120 real-world news articles on Apple’s Tier 1 suppliers, we adopt a standardized, prompt-based assessment to compare GPT-3.5 turbo, GPT-4o, GPT-4o mini, Claude 3.5 Sonnet, and Claude 3.5 Haiku. Using expert-reviewed metrics, namely the Risk Validation Rate (RVR), Potential Risk Rate (PRR), and False Identification Rate (FIR), we derive a comprehensive Relative Performance Index (RPI) for comparison. Our analysis confirms that advanced GPT-4o variants produce the most consistent accurate risk identifications, achieving higher proportions of validated outcomes while minimizing false positives. Through these results, we highlight the significant promise of LLM-driven analytics for early risk detection in complex supply chains, along with practical considerations such as the influence of prompt engineering, interpretability demands, and the impact of data availability. The findings offer a blueprint for organizations seeking to improve resilience by systematically harnessing the capabilities of LLM within service-oriented risk management ecosystems.

Please use this identifier to cite or link to this item:

http://hdl.handle.net/10453/192545