Soft Prompt Transfer for Zero-Shot and Few-Shot Learning in EHR Understanding

Wang, Y; Peng, X; Shen, T; Clarke, A; Schlegel, C; Martin, P; Long, G

Soft Prompt Transfer for Zero-Shot and Few-Shot Learning in EHR Understanding

Wang, Y Peng, X

Shen, T Clarke, A Schlegel, C Martin, P Long, G

Permalink

Publisher:: Springer Nature
Publication Type:: Chapter
Citation:: Advanced Data Mining and Applications, 2023, 14178 LNAI, pp. 18-32
Issue Date:: 2023-01-01

Closed Access

	Filename	Description	Size
	978-3-031-46671-7_2.pdf		418.89 kB	Adobe PDF	View/Open

Copyright Clearance Process

Recently Added
In Progress
Closed Access

This item is closed access and not available.

Full metadata record

Field	Value	Language
dc.contributor.author	Wang, Y
dc.contributor.author	Peng, X https://orcid.org/0000-0002-8901-1472
dc.contributor.author	Shen, T
dc.contributor.author	Clarke, A
dc.contributor.author	Schlegel, C
dc.contributor.author	Martin, P
dc.contributor.author	Long, G https://orcid.org/0000-0003-3740-9515
dc.date.accessioned	2023-12-04T02:30:42Z
dc.date.available	2023-12-04T02:30:42Z
dc.date.issued	2023-01-01
dc.identifier.citation	Advanced Data Mining and Applications, 2023, 14178 LNAI, pp. 18-32
dc.identifier.isbn	9783031466700
dc.identifier.uri	http://hdl.handle.net/10453/173655
dc.description.abstract	Electronic Health Records (EHRs) are a rich source of information that can be leveraged for various medical applications, such as disease inference, treatment recommendation, and outcome analysis. However, the complexity and heterogeneity of EHR data, along with the limited availability of well-labeled samples, present significant challenges to the development of efficient and adaptable models for EHR tasks (such as rare or novel disease prediction or inference). In this paper, we propose Soft prompt transfer for Electronic Health Records (SptEHR), a novel pipeline designed to address these challenges. Specifically, SptEHR consists of three main stages: (1) self-supervised pre-training on raw EHR data for an EHR-centric transformer-based foundation model, (2) supervised multi-task continual learning from existing well-labeled tasks to further refine the foundation model and learn transferable task-specific soft prompts, and (3) further improve zero-shot and few-shot ability via prompt transfer. Specifically, the transformer-based foundation model learned from stage one captures domain-specific knowledge. Then the multi-task continual training in stage two improves model adaptability and performance on EHR tasks. Finally, stage three leverages soft prompt transfer which is based on the similarity between the new and the existing tasks, to effectively address new tasks without requiring additional/extensive training. The effectiveness of the SptEHR has been validated on the benchmark dataset - MIMIC-III.
dc.language	en
dc.publisher	Springer Nature
dc.relation.ispartof	Advanced Data Mining and Applications
dc.relation.ispartofseries	Lecture Notes in Computer Science
dc.relation.isbasedon	10.1007/978-3-031-46671-7_2
dc.rights	info:eu-repo/semantics/closedAccess
dc.subject.classification	Artificial Intelligence & Image Processing
dc.subject.classification	46 Information and computing sciences
dc.title	Soft Prompt Transfer for Zero-Shot and Few-Shot Learning in EHR Understanding
dc.type	Chapter
utslib.citation.volume	14178 LNAI
pubs.organisational-group	/University of Technology Sydney
pubs.organisational-group	/University of Technology Sydney/Faculty of Engineering and Information Technology
pubs.organisational-group	/University of Technology Sydney/Strength - AAII - Australian Artificial Intelligence Institute
pubs.organisational-group	/University of Technology Sydney/Faculty of Engineering and Information Technology/School of Computer Science
utslib.copyright.status	closed_access	*
dc.date.updated	2023-12-04T02:30:40Z
pubs.publication-status	Published
pubs.volume	14178 LNAI

Abstract:

Electronic Health Records (EHRs) are a rich source of information that can be leveraged for various medical applications, such as disease inference, treatment recommendation, and outcome analysis. However, the complexity and heterogeneity of EHR data, along with the limited availability of well-labeled samples, present significant challenges to the development of efficient and adaptable models for EHR tasks (such as rare or novel disease prediction or inference). In this paper, we propose Soft prompt transfer for Electronic Health Records (SptEHR), a novel pipeline designed to address these challenges. Specifically, SptEHR consists of three main stages: (1) self-supervised pre-training on raw EHR data for an EHR-centric transformer-based foundation model, (2) supervised multi-task continual learning from existing well-labeled tasks to further refine the foundation model and learn transferable task-specific soft prompts, and (3) further improve zero-shot and few-shot ability via prompt transfer. Specifically, the transformer-based foundation model learned from stage one captures domain-specific knowledge. Then the multi-task continual training in stage two improves model adaptability and performance on EHR tasks. Finally, stage three leverages soft prompt transfer which is based on the similarity between the new and the existing tasks, to effectively address new tasks without requiring additional/extensive training. The effectiveness of the SptEHR has been validated on the benchmark dataset - MIMIC-III.

Please use this identifier to cite or link to this item:

http://hdl.handle.net/10453/173655