Noisy Correspondence Learning with Self-Reinforcing Errors Mitigation
- Publisher:
- ASSOC ADVANCEMENT ARTIFICIAL INTELLIGENCE
- Publication Type:
- Conference Proceeding
- Citation:
- Proceedings of the AAAI Conference on Artificial Intelligence, 2024, 38, (2), pp. 1463-1471
- Issue Date:
- 2024-03-25
Closed Access
| Filename | Description | Size | |||
|---|---|---|---|---|---|
| 2312.16478v1.pdf | Published version | 5.96 MB |
Copyright Clearance Process
- Recently Added
- In Progress
- Closed Access
This item is closed access and not available.
Cross-modal retrieval relies on well-matched large-scale datasets that are laborious in practice. Recently, to alleviate expensive data collection, co-occurring pairs from the Internet are automatically harvested for training. However, it inevitably includes mismatched pairs, i.e., noisy correspondences, undermining supervision reliability and degrading performance. Current methods leverage deep neural networks’ memorization effect to address noisy correspondences, which overconfidently focus on similarity-guided training with hard negatives and suffer from self-reinforcing errors. In light of above, we introduce a novel noisy correspondence learning framework, namely Self-Reinforcing Errors Mitigation (SREM). Specifically, by viewing sample matching as classification tasks within the batch, we generate classification logits for the given sample. Instead of a single similarity score, we refine sample filtration through energy uncertainty and estimate model’s sensitivity of selected clean samples using swapped classification entropy, in view of the overall prediction distribution. Additionally, we propose cross-modal biased complementary learning to leverage negative matches overlooked in hard-negative training, further improving model optimization stability and curbing self-reinforcing errors. Extensive experiments on challenging benchmarks affirm the efficacy and efficiency of SREM.
Please use this identifier to cite or link to this item: