Semantic Correspondence as an Optimal Transport Problem

Liu, Y; Zhu, L; Yamada, M; Yang, Y

Semantic Correspondence as an Optimal Transport Problem

Liu, Y Zhu, L

Yamada, M Yang, Y

Permalink

Publisher:: IEEE
Publication Type:: Conference Proceeding
Citation:: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020, 00, pp. 4462-4471
Issue Date:: 2020-08-05

Closed Access

	Filename	Description	Size
	09156442.pdf	Published version	877.87 kB	Adobe PDF	View/Open

Copyright Clearance Process

Recently Added
In Progress
Closed Access

This item is closed access and not available.

Full metadata record

Field	Value	Language
dc.contributor.author	Liu, Y
dc.contributor.author	Zhu, L https://orcid.org/0000-0002-4093-7557
dc.contributor.author	Yamada, M
dc.contributor.author	Yang, Y https://orcid.org/0000-0002-0512-880X
dc.date	2020-06-13
dc.date.accessioned	2021-04-15T10:27:13Z
dc.date.available	2021-04-15T10:27:13Z
dc.date.issued	2020-08-05
dc.identifier.citation	2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020, 00, pp. 4462-4471
dc.identifier.isbn	978-1-7281-7169-2
dc.identifier.issn	1063-6919
dc.identifier.uri	http://hdl.handle.net/10453/148144
dc.description.abstract	Establishing dense correspondences across semantically similar images is a challenging task. Due to the large intra-class variation and background clutter, two common issues occur in current approaches. First, many pixels in a source image are assigned to one target pixel, i.e., many to one matching. Second, some object pixels are assigned to the background pixels, i.e., background matching. We solve the first issue by global feature matching, which maximizes the total matching correlations between images to obtain a global optimal matching matrix. The row sum and column sum constraints are enforced on the matching matrix to induce a balanced solution, thus suppressing the many to one matching. We solve the second issue by applying a staircase function on the class activation maps to re-weight the importance of pixels into four levels from foreground to background. The whole procedure is combined into a unified optimal transport algorithm by converting the maximization problem to the optimal transport formulation and incorporating the staircase weights into optimal transport algorithm to act as empirical distributions. The proposed algorithm achieves state-of-the-art performance on four benchmark datasets. Notably, a 26\% relative improvement is achieved on the large-scale SPair-71k dataset.
dc.language	en
dc.publisher	IEEE
dc.relation	http://purl.org/au-research/grants/arc/DP200100938
dc.relation.ispartof	2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
dc.relation.ispartof	2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition
dc.relation.ispartofseries	IEEE Conference on Computer Vision and Pattern Recognition
dc.relation.isbasedon	10.1109/cvpr42600.2020.00452
dc.rights	info:eu-repo/semantics/closedAccess
dc.title	Semantic Correspondence as an Optimal Transport Problem
dc.type	Conference Proceeding
utslib.citation.volume	00
utslib.location.activity	Seattle, WA, USA
utslib.for	0801 Artificial Intelligence and Image Processing
pubs.organisational-group	/University of Technology Sydney
pubs.organisational-group	/University of Technology Sydney/Faculty of Engineering and Information Technology
pubs.organisational-group	/University of Technology Sydney/Strength - AAII - Australian Artificial Intelligence Institute
pubs.organisational-group	/University of Technology Sydney/Faculty of Engineering and Information Technology/School of Computer Science
utslib.copyright.status	closed_access	*
dc.date.updated	2021-04-15T10:27:12Z
pubs.finish-date	2020-06-19
pubs.publication-status	Published
pubs.start-date	2020-06-13
pubs.volume	00

Abstract:

Establishing dense correspondences across semantically similar images is a challenging task. Due to the large intra-class variation and background clutter, two common issues occur in current approaches. First, many pixels in a source image are assigned to one target pixel, i.e., many to one matching. Second, some object pixels are assigned to the background pixels, i.e., background matching. We solve the first issue by global feature matching, which maximizes the total matching correlations between images to obtain a global optimal matching matrix. The row sum and column sum constraints are enforced on the matching matrix to induce a balanced solution, thus suppressing the many to one matching. We solve the second issue by applying a staircase function on the class activation maps to re-weight the importance of pixels into four levels from foreground to background. The whole procedure is combined into a unified optimal transport algorithm by converting the maximization problem to the optimal transport formulation and incorporating the staircase weights into optimal transport algorithm to act as empirical distributions. The proposed algorithm achieves state-of-the-art performance on four benchmark datasets. Notably, a 26\% relative improvement is achieved on the large-scale SPair-71k dataset.

Please use this identifier to cite or link to this item:

http://hdl.handle.net/10453/148144