Is Heterogeneity Notorious? Taming Heterogeneity to Handle Test-Time Shift in Federated Learning

Tan, Y; Chen, C; Zhuang, W; Dong, X; Lyu, L; Long, G

Is Heterogeneity Notorious? Taming Heterogeneity to Handle Test-Time Shift in Federated Learning

Tan, Y Chen, C Zhuang, W Dong, X Lyu, L Long, G

Permalink

Publication Type:: Conference Proceeding
Citation:: Advances in Neural Information Processing Systems, 2023, 36
Issue Date:: 2023-01-01

Open Access

Copyright Clearance Process

Recently Added
In Progress
Open Access

This item is open access.

Adobe PDF

Download Published versionAdobe PDF (1.06 MB)

View statistics

Full metadata record

Field	Value	Language
dc.contributor.author	Tan, Y
dc.contributor.author	Chen, C
dc.contributor.author	Zhuang, W
dc.contributor.author	Dong, X
dc.contributor.author	Lyu, L
dc.contributor.author	Long, G https://orcid.org/0000-0003-3740-9515
dc.date	2023-12-10
dc.date.accessioned	2024-06-19T02:58:06Z
dc.date.available	2024-06-19T02:58:06Z
dc.date.issued	2023-01-01
dc.identifier.citation	Advances in Neural Information Processing Systems, 2023, 36
dc.identifier.issn	1049-5258
dc.identifier.uri	http://hdl.handle.net/10453/179569
dc.description.abstract	Federated learning (FL) is an effective machine learning paradigm where multiple clients can train models based on heterogeneous data in a decentralized manner without accessing their private data. However, existing FL systems undergo performance deterioration due to feature-level test-time shifts, which are well investigated in centralized settings but rarely studied in FL. The common non-IID issue in FL usually refers to inter-client heterogeneity during training phase, while the test-time shift refers to the intra-client heterogeneity during test phase. Although the former is always deemed to be notorious for FL, there is still a wealth of useful information delivered by heterogeneous data sources, which may potentially help alleviate the latter issue. To explore the possibility of using inter-client heterogeneity in handling intra-client heterogeneity, we firstly propose a contrastive learning-based FL framework, namely FedICON, to capture invariant knowledge among heterogeneous clients and consistently tune the model to adapt to test data. In FedICON, each client performs sample-wise supervised contrastive learning during the local training phase, which enhances sample-wise invariance encoding ability. Through global aggregation, the invariance extraction ability can be mutually boosted among inter-client heterogeneity. During the test phase, our test-time adaptation procedure leverages unsupervised contrastive learning to guide the model to smoothly generalize to test data under intra-client heterogeneity. Extensive experiments validate the effectiveness of the proposed FedICON in taming heterogeneity to handle test-time shift problems.
dc.language	en
dc.relation.ispartof	Advances in Neural Information Processing Systems
dc.relation.ispartof	Conference on Neural Information Processing Systems
dc.rights	info:eu-repo/semantics/openAccess
dc.subject	1701 Psychology, 1702 Cognitive Sciences
dc.subject.classification	4611 Machine learning
dc.title	Is Heterogeneity Notorious? Taming Heterogeneity to Handle Test-Time Shift in Federated Learning
dc.type	Conference Proceeding
utslib.citation.volume	36
utslib.location.activity	New Orleans, USA
utslib.for	1701 Psychology
utslib.for	1702 Cognitive Sciences
pubs.organisational-group	University of Technology Sydney
pubs.organisational-group	University of Technology Sydney/Faculty of Engineering and Information Technology
pubs.organisational-group	University of Technology Sydney/Strength - AAII - Australian Artificial Intelligence Institute
utslib.copyright.status	open_access	*
pubs.consider-herdc	false
dc.date.updated	2024-06-19T02:58:05Z
pubs.finish-date	2023-12-16
pubs.publication-status	Published
pubs.start-date	2023-12-10
pubs.volume	36

Abstract:

Federated learning (FL) is an effective machine learning paradigm where multiple clients can train models based on heterogeneous data in a decentralized manner without accessing their private data. However, existing FL systems undergo performance deterioration due to feature-level test-time shifts, which are well investigated in centralized settings but rarely studied in FL. The common non-IID issue in FL usually refers to inter-client heterogeneity during training phase, while the test-time shift refers to the intra-client heterogeneity during test phase. Although the former is always deemed to be notorious for FL, there is still a wealth of useful information delivered by heterogeneous data sources, which may potentially help alleviate the latter issue. To explore the possibility of using inter-client heterogeneity in handling intra-client heterogeneity, we firstly propose a contrastive learning-based FL framework, namely FedICON, to capture invariant knowledge among heterogeneous clients and consistently tune the model to adapt to test data. In FedICON, each client performs sample-wise supervised contrastive learning during the local training phase, which enhances sample-wise invariance encoding ability. Through global aggregation, the invariance extraction ability can be mutually boosted among inter-client heterogeneity. During the test phase, our test-time adaptation procedure leverages unsupervised contrastive learning to guide the model to smoothly generalize to test data under intra-client heterogeneity. Extensive experiments validate the effectiveness of the proposed FedICON in taming heterogeneity to handle test-time shift problems.

Please use this identifier to cite or link to this item:

http://hdl.handle.net/10453/179569