From Transcripts to Themes: A Trustworthy Workflow for Qualitative Analysis Using Large Language Models

Bakharia, A; Shibani, A; Lim, LA; McCluskey, T; Shum, SB

From Transcripts to Themes: A Trustworthy Workflow for Qualitative Analysis Using Large Language Models

Bakharia, A Shibani, A Lim, LA McCluskey, T Shum, SB

Permalink

Publication Type:: Conference Proceeding
Citation:: Ceur Workshop Proceedings, 2025, 3995, pp. 179-189
Issue Date:: 2025-01-01

Open Access

Copyright Clearance Process

Recently Added
In Progress
Open Access

This item is open access.

Adobe PDF

Download Published versionAdobe PDF (1.36 MB)

View statistics

Full metadata record

Field	Value	Language
dc.contributor.author	Bakharia, A
dc.contributor.author	Shibani, A
dc.contributor.author	Lim, LA
dc.contributor.author	McCluskey, T
dc.contributor.author	Shum, SB
dc.date.accessioned	2025-12-18T08:35:24Z
dc.date.available	2025-12-18T08:35:24Z
dc.date.issued	2025-01-01
dc.identifier.citation	Ceur Workshop Proceedings, 2025, 3995, pp. 179-189
dc.identifier.issn	1613-0073
dc.identifier.uri	http://hdl.handle.net/10453/190990
dc.description.abstract	We present a novel workflow that leverages Large Language Models (LLMs) to advance qualitative analysis within Learning Analytics, addressing the limitations of existing approaches that fall short in providing theme labels, hierarchical categorization, and supporting evidence, creating a gap in effective sensemaking of learner-generated data. Our approach uses LLMs for inductive analysis from open text, enabling the extraction and description of themes with supporting quotes and hierarchical categories. This trustworthy workflow allows for researcher review and input at every stage, ensuring traceability and verification, key requirements for qualitative analysis. Applied to a focus group dataset on student perspectives on generative AI in higher education, our method demonstrates that LLMs are able to effectively extract quotes and provide labeled interpretable themes compared to traditional topic modeling algorithms. Our proposed workflow provides comprehensive insights into learner behaviors and experiences and offers educators an additional lens to understand and categorize student-generated data according to deeper learning constructs, which can facilitate richer and more actionable insights for Learning Analytics.
dc.language	en
dc.relation.ispartof	Ceur Workshop Proceedings
dc.rights	info:eu-repo/semantics/openAccess
dc.subject.classification	4609 Information systems
dc.title	From Transcripts to Themes: A Trustworthy Workflow for Qualitative Analysis Using Large Language Models
dc.type	Conference Proceeding
utslib.citation.volume	3995
pubs.organisational-group	University of Technology Sydney
pubs.organisational-group	University of Technology Sydney/Provost
pubs.organisational-group	University of Technology Sydney/Provost/TD School
pubs.organisational-group	University of Technology Sydney/UTS Groups
pubs.organisational-group	University of Technology Sydney/UTS Groups/The Trustworthy Digital Society
utslib.copyright.status	open_access	*
dc.rights.license	This work is licensed under a Creative Commons Attribution 4.0 International License (CC BY 4.0). To view a copy of this license, visit https://creativecommons.org/licenses/by/4.0/
dc.date.updated	2025-12-18T08:35:22Z
pubs.publication-status	Published
pubs.volume	3995

Abstract:

We present a novel workflow that leverages Large Language Models (LLMs) to advance qualitative analysis within Learning Analytics, addressing the limitations of existing approaches that fall short in providing theme labels, hierarchical categorization, and supporting evidence, creating a gap in effective sensemaking of learner-generated data. Our approach uses LLMs for inductive analysis from open text, enabling the extraction and description of themes with supporting quotes and hierarchical categories. This trustworthy workflow allows for researcher review and input at every stage, ensuring traceability and verification, key requirements for qualitative analysis. Applied to a focus group dataset on student perspectives on generative AI in higher education, our method demonstrates that LLMs are able to effectively extract quotes and provide labeled interpretable themes compared to traditional topic modeling algorithms. Our proposed workflow provides comprehensive insights into learner behaviors and experiences and offers educators an additional lens to understand and categorize student-generated data according to deeper learning constructs, which can facilitate richer and more actionable insights for Learning Analytics.

Please use this identifier to cite or link to this item:

http://hdl.handle.net/10453/190990