SGINet: Toward Sufficient Interaction Between Single Image Deraining and Semantic Segmentation

Wei, Y; Zhang, Z; Zheng, H; Hong, R; Yang, Y; Wang, M

SGINet: Toward Sufficient Interaction Between Single Image Deraining and Semantic Segmentation

Wei, Y Zhang, Z Zheng, H Hong, R Yang, Y

Wang, M

Permalink

Publisher:: Association for Computing Machinery (ACM)
Publication Type:: Conference Proceeding
Citation:: MM 2022 - Proceedings of the 30th ACM International Conference on Multimedia, 2022, pp. 6202-6210
Issue Date:: 2022-10-10

Closed Access

	Filename	Description	Size
	SGINet.pdf	Published version	4.26 MB	Adobe PDF	View/Open

Copyright Clearance Process

Recently Added
In Progress
Closed Access

This item is closed access and not available.

Full metadata record

Field	Value	Language
dc.contributor.author	Wei, Y
dc.contributor.author	Zhang, Z
dc.contributor.author	Zheng, H
dc.contributor.author	Hong, R
dc.contributor.author	Yang, Y https://orcid.org/0000-0002-0512-880X
dc.contributor.author	Wang, M
dc.date.accessioned	2023-05-19T06:24:59Z
dc.date.available	2023-05-19T06:24:59Z
dc.date.issued	2022-10-10
dc.identifier.citation	MM 2022 - Proceedings of the 30th ACM International Conference on Multimedia, 2022, pp. 6202-6210
dc.identifier.isbn	9781450392037
dc.identifier.uri	http://hdl.handle.net/10453/170377
dc.description.abstract	Data-driven single image deraining (SID) models have achieved greater progress by simulations, but there is still a large gap between current deraining performance and practical high-level applications, since high-level semantic information is usually neglected in current studies. Although few studies jointly considered high-level tasks (e.g., segmentation) to enable the model to learn more high-level information, there are two obvious shortcomings. First, they require the segmentation labels for training, limiting their operations on other datasets without high-level labels. Second, high- and low-level information are not fully interacted, hence having limited improvement in both deraining and segmentation tasks. In this paper, we propose a Semantic Guided Interactive Network (SGINet), which considers the sufficient interaction between SID and semantic segmentation using a three-stage deraining manner, i.e., coarse deraining, semantic information extraction, and semantics guided deraining. Specifically, a Full Resolution Module (FRM) without down-/up-sampling is proposed to predict the coarse deraining images without context damage. Then, a Segmentation Extracting Module (SEM) is designed to extract accurate semantic information. We also develop a novel contrastive semantic discovery (CSD) loss, which can instruct the process of semantic segmentation without real semantic segmentation labels. Finally, a triple-direction U-net-based Semantic Interaction Module (SIM) takes advantage of the coarse deraining images and semantic information for fully interacting low-level with high-level tasks. Extensive simulations on the newly-constructed complex datasets Cityscapes_syn and Cityscapes_real demonstrated that our model could obtain more promising results. Overall, our SGINet achieved SOTA deraining and segmentation performance in both simulation and real-scenario data, compared with other representative SID methods.
dc.language	en
dc.publisher	Association for Computing Machinery (ACM)
dc.relation.ispartof	MM 2022 - Proceedings of the 30th ACM International Conference on Multimedia
dc.relation.ispartof	Proceedings of the 30th ACM International Conference on Multimedia
dc.relation.isbasedon	10.1145/3503161.3548241
dc.rights	info:eu-repo/semantics/closedAccess
dc.title	SGINet: Toward Sufficient Interaction Between Single Image Deraining and Semantic Segmentation
dc.type	Conference Proceeding
pubs.organisational-group	/University of Technology Sydney
pubs.organisational-group	/University of Technology Sydney/Faculty of Engineering and Information Technology
utslib.copyright.status	closed_access	*
dc.date.updated	2023-05-19T06:24:56Z
pubs.publication-status	Published

Abstract:

Data-driven single image deraining (SID) models have achieved greater progress by simulations, but there is still a large gap between current deraining performance and practical high-level applications, since high-level semantic information is usually neglected in current studies. Although few studies jointly considered high-level tasks (e.g., segmentation) to enable the model to learn more high-level information, there are two obvious shortcomings. First, they require the segmentation labels for training, limiting their operations on other datasets without high-level labels. Second, high- and low-level information are not fully interacted, hence having limited improvement in both deraining and segmentation tasks. In this paper, we propose a Semantic Guided Interactive Network (SGINet), which considers the sufficient interaction between SID and semantic segmentation using a three-stage deraining manner, i.e., coarse deraining, semantic information extraction, and semantics guided deraining. Specifically, a Full Resolution Module (FRM) without down-/up-sampling is proposed to predict the coarse deraining images without context damage. Then, a Segmentation Extracting Module (SEM) is designed to extract accurate semantic information. We also develop a novel contrastive semantic discovery (CSD) loss, which can instruct the process of semantic segmentation without real semantic segmentation labels. Finally, a triple-direction U-net-based Semantic Interaction Module (SIM) takes advantage of the coarse deraining images and semantic information for fully interacting low-level with high-level tasks. Extensive simulations on the newly-constructed complex datasets Cityscapes_syn and Cityscapes_real demonstrated that our model could obtain more promising results. Overall, our SGINet achieved SOTA deraining and segmentation performance in both simulation and real-scenario data, compared with other representative SID methods.

Please use this identifier to cite or link to this item:

http://hdl.handle.net/10453/170377