Customized meta-dataset for automatic classifier accuracy evaluation

Huang, Y; Zhang, Z; Wu, Q; Huang, H; Zhong, Y; Wang, L

Customized meta-dataset for automatic classifier accuracy evaluation

Huang, Y Zhang, Z Wu, Q

Huang, H Zhong, Y Wang, L

Permalink

Publisher:: ELSEVIER SCI LTD
Publication Type:: Journal Article
Citation:: Pattern Recognition, 2024, 146
Issue Date:: 2024-02-01

Closed Access

	Filename	Description	Size
	1-s2.0-S0031320323007239-main.pdf	Published version	1.7 MB	Adobe PDF	View/Open

Copyright Clearance Process

Recently Added
In Progress
Closed Access

This item is closed access and not available.

Full metadata record

Field	Value	Language
dc.contributor.author	Huang, Y
dc.contributor.author	Zhang, Z
dc.contributor.author	Wu, Q https://orcid.org/0000-0001-5641-2483
dc.contributor.author	Huang, H
dc.contributor.author	Zhong, Y
dc.contributor.author	Wang, L
dc.date.accessioned	2024-03-13T10:11:30Z
dc.date.available	2024-03-13T10:11:30Z
dc.date.issued	2024-02-01
dc.identifier.citation	Pattern Recognition, 2024, 146
dc.identifier.issn	0031-3203
dc.identifier.issn	1873-5142
dc.identifier.uri	http://hdl.handle.net/10453/176637
dc.description.abstract	Automatic classifier accuracy evaluation (ACAEval) on unlabeled test sets is critical for unseen real-world environments. The use of dataset-level regression on synthesized meta-datasets (comprised of many sample sets) has shown promising results for ACAEval. However, the existing meta-dataset for ACAEval is created using simple image transformations such as rotation and background substitution, which can make it difficult to ensure a reasonable distribution shift between the sample set and the test set. When the distribution shift is large, it becomes challenging to estimate the classifier accuracy on the test set using those sample sets. To ensure more robust ACAEval, this paper attempts to customize a meta-dataset in which each sample set has a reasonable distribution shift to the test set. An intra-class cycle-consistent adversarial learning (ICAL) method is introduced to transfer the style of a labeled training set to the style of the test set, by jointly considering the domain shift issue, the label flipping issue (the semantic information may be changed after style transformation), and the diversity of multiple sample sets in the meta-dataset. Experiments validate that under the same experimental setup, our method outperforms the existing ACAEval methods by a good margin, and achieves state-of-the-art performance on several standard benchmark datasets, including digit classification and natural image classification.
dc.language	English
dc.publisher	ELSEVIER SCI LTD
dc.relation.ispartof	Pattern Recognition
dc.relation.isbasedon	10.1016/j.patcog.2023.110026
dc.rights	info:eu-repo/semantics/closedAccess
dc.subject	0801 Artificial Intelligence and Image Processing, 0806 Information Systems, 0906 Electrical and Electronic Engineering
dc.subject.classification	Artificial Intelligence & Image Processing
dc.subject.classification	4603 Computer vision and multimedia computation
dc.subject.classification	4605 Data management and data science
dc.subject.classification	4611 Machine learning
dc.title	Customized meta-dataset for automatic classifier accuracy evaluation
dc.type	Journal Article
utslib.citation.volume	146
utslib.for	0801 Artificial Intelligence and Image Processing
utslib.for	0806 Information Systems
utslib.for	0906 Electrical and Electronic Engineering
pubs.organisational-group	University of Technology Sydney
pubs.organisational-group	University of Technology Sydney/Faculty of Engineering and Information Technology
pubs.organisational-group	University of Technology Sydney/Strength - INEXT - Innovation in IT Services and Applications
pubs.organisational-group	University of Technology Sydney/Strength - GBDTC - Global Big Data Technologies
pubs.organisational-group	University of Technology Sydney/Faculty of Engineering and Information Technology/School of Electrical and Data Engineering
utslib.copyright.status	closed_access	*
dc.date.updated	2024-03-13T10:11:28Z
pubs.publication-status	Published
pubs.volume	146

Abstract:

Automatic classifier accuracy evaluation (ACAEval) on unlabeled test sets is critical for unseen real-world environments. The use of dataset-level regression on synthesized meta-datasets (comprised of many sample sets) has shown promising results for ACAEval. However, the existing meta-dataset for ACAEval is created using simple image transformations such as rotation and background substitution, which can make it difficult to ensure a reasonable distribution shift between the sample set and the test set. When the distribution shift is large, it becomes challenging to estimate the classifier accuracy on the test set using those sample sets. To ensure more robust ACAEval, this paper attempts to customize a meta-dataset in which each sample set has a reasonable distribution shift to the test set. An intra-class cycle-consistent adversarial learning (ICAL) method is introduced to transfer the style of a labeled training set to the style of the test set, by jointly considering the domain shift issue, the label flipping issue (the semantic information may be changed after style transformation), and the diversity of multiple sample sets in the meta-dataset. Experiments validate that under the same experimental setup, our method outperforms the existing ACAEval methods by a good margin, and achieves state-of-the-art performance on several standard benchmark datasets, including digit classification and natural image classification.

Please use this identifier to cite or link to this item:

http://hdl.handle.net/10453/176637