Do comprehensive deep learning algorithms suffer from hidden stratification? A retrospective study on pneumothorax detection in chest radiography

Seah, J; Tang, C; Buchlak, QD; Milne, MR; Holt, X; Ahmad, H; Lambert, J; Esmaili, N; Oakden-Rayner, L; Brotchie, P; Jones, CM

Do comprehensive deep learning algorithms suffer from hidden stratification? A retrospective study on pneumothorax detection in chest radiography

Seah, J Tang, C Buchlak, QD Milne, MR Holt, X Ahmad, H Lambert, J Esmaili, N Oakden-Rayner, L Brotchie, P Jones, CM

Permalink

Publisher:: BMJ
Publication Type:: Journal Article
Citation:: BMJ Open, 2021, 11, (12), pp. e053024
Issue Date:: 2021-12-07

Open Access

Copyright Clearance Process

Recently Added
In Progress
Open Access

This item is open access.

Adobe PDF

Download Published versionAdobe PDF (492.14 kB)

View on publisher's site

View statistics

Full metadata record

Field	Value	Language
dc.contributor.author	Seah, J
dc.contributor.author	Tang, C
dc.contributor.author	Buchlak, QD
dc.contributor.author	Milne, MR
dc.contributor.author	Holt, X
dc.contributor.author	Ahmad, H
dc.contributor.author	Lambert, J
dc.contributor.author	Esmaili, N
dc.contributor.author	Oakden-Rayner, L
dc.contributor.author	Brotchie, P
dc.contributor.author	Jones, CM
dc.date.accessioned	2022-01-13T02:25:46Z
dc.date.available	2022-01-13T02:25:46Z
dc.date.issued	2021-12-07
dc.identifier.citation	BMJ Open, 2021, 11, (12), pp. e053024
dc.identifier.issn	2044-6055
dc.identifier.issn	2044-6055
dc.identifier.uri	http://hdl.handle.net/10453/153032
dc.description.abstract	<jats:sec><jats:title>Objectives</jats:title><jats:p>To evaluate the ability of a commercially available comprehensive chest radiography deep convolutional neural network (DCNN) to detect simple and tension pneumothorax, as stratified by the following subgroups: the presence of an intercostal drain; rib, clavicular, scapular or humeral fractures or rib resections; subcutaneous emphysema and erect versus non-erect positioning. The hypothesis was that performance would not differ significantly in each of these subgroups when compared with the overall test dataset.</jats:p></jats:sec><jats:sec><jats:title>Design</jats:title><jats:p>A retrospective case–control study was undertaken.</jats:p></jats:sec><jats:sec><jats:title>Setting</jats:title><jats:p>Community radiology clinics and hospitals in Australia and the USA.</jats:p></jats:sec><jats:sec><jats:title>Participants</jats:title><jats:p>A test dataset of 2557 chest radiography studies was ground-truthed by three subspecialty thoracic radiologists for the presence of simple or tension pneumothorax as well as each subgroup other than positioning. Radiograph positioning was derived from radiographer annotations on the images.</jats:p></jats:sec><jats:sec><jats:title>Outcome measures</jats:title><jats:p>DCNN performance for detecting simple and tension pneumothorax was evaluated over the entire test set, as well as within each subgroup, using the area under the receiver operating characteristic curve (AUC). A difference in AUC of more than 0.05 was considered clinically significant.</jats:p></jats:sec><jats:sec><jats:title>Results</jats:title><jats:p>When compared with the overall test set, performance of the DCNN for detecting simple and tension pneumothorax was statistically non-inferior in all subgroups. The DCNN had an AUC of 0.981 (0.976–0.986) for detecting simple pneumothorax and 0.997 (0.995–0.999) for detecting tension pneumothorax.</jats:p></jats:sec><jats:sec><jats:title>Conclusions</jats:title><jats:p>Hidden stratification has significant implications for potential failures of deep learning when applied in clinical practice. This study demonstrated that a comprehensively trained DCNN can be resilient to hidden stratification in several clinically meaningful subgroups in detecting pneumothorax.</jats:p></jats:sec>
dc.language	en
dc.publisher	BMJ
dc.relation.ispartof	BMJ Open
dc.relation.isbasedon	10.1136/bmjopen-2021-053024
dc.rights	info:eu-repo/semantics/openAccess
dc.subject	1103 Clinical Sciences, 1117 Public Health and Health Services, 1199 Other Medical and Health Sciences
dc.title	Do comprehensive deep learning algorithms suffer from hidden stratification? A retrospective study on pneumothorax detection in chest radiography
dc.type	Journal Article
utslib.citation.volume	11
utslib.for	1103 Clinical Sciences
utslib.for	1117 Public Health and Health Services
utslib.for	1199 Other Medical and Health Sciences
pubs.organisational-group	/University of Technology Sydney
pubs.organisational-group	/University of Technology Sydney/Faculty of Engineering and Information Technology
pubs.organisational-group	/University of Technology Sydney/Faculty of Engineering and Information Technology/School of Electrical and Data Engineering
utslib.copyright.status	open_access	*
dc.date.updated	2022-01-13T02:25:45Z
pubs.issue	12
pubs.publication-status	Published
pubs.volume	11
utslib.citation.issue	12

Abstract:

ObjectivesTo evaluate the ability of a commercially available comprehensive chest radiography deep convolutional neural network (DCNN) to detect simple and tension pneumothorax, as stratified by the following subgroups: the presence of an intercostal drain; rib, clavicular, scapular or humeral fractures or rib resections; subcutaneous emphysema and erect versus non-erect positioning. The hypothesis was that performance would not differ significantly in each of these subgroups when compared with the overall test dataset.DesignA retrospective case–control study was undertaken.SettingCommunity radiology clinics and hospitals in Australia and the USA.ParticipantsA test dataset of 2557 chest radiography studies was ground-truthed by three subspecialty thoracic radiologists for the presence of simple or tension pneumothorax as well as each subgroup other than positioning. Radiograph positioning was derived from radiographer annotations on the images.Outcome measuresDCNN performance for detecting simple and tension pneumothorax was evaluated over the entire test set, as well as within each subgroup, using the area under the receiver operating characteristic curve (AUC). A difference in AUC of more than 0.05 was considered clinically significant.ResultsWhen compared with the overall test set, performance of the DCNN for detecting simple and tension pneumothorax was statistically non-inferior in all subgroups. The DCNN had an AUC of 0.981 (0.976–0.986) for detecting simple pneumothorax and 0.997 (0.995–0.999) for detecting tension pneumothorax.ConclusionsHidden stratification has significant implications for potential failures of deep learning when applied in clinical practice. This study demonstrated that a comprehensively trained DCNN can be resilient to hidden stratification in several clinically meaningful subgroups in detecting pneumothorax.

Please use this identifier to cite or link to this item:

http://hdl.handle.net/10453/153032