Leveraging Weak Semantic Relevance for Complex Video Event Classification

Shen, HT; Li, C; Cao, J; Huang, Z; Zhu, L

Leveraging Weak Semantic Relevance for Complex Video Event Classification

Shen, HT Li, C Cao, J Huang, Z Zhu, L

Permalink

Publication Type:: Conference Proceeding
Citation:: Proceedings of the IEEE International Conference on Computer Vision, 2017, 2017-October pp. 3667 - 3676
Issue Date:: 2017-12-22

Closed Access

	Filename	Description	Size
	08237656.pdf	Published version	440.82 kB	Adobe PDF	View/Open

Copyright Clearance Process

Recently Added
In Progress
Closed Access

This item is closed access and not available.

Full metadata record

Field	Value	Language
dc.contributor.author	Shen, HT	en_US
dc.contributor.author	Li, C	en_US
dc.contributor.author	Cao, J	en_US
dc.contributor.author	Huang, Z	en_US
dc.contributor.author	Zhu, L	en_US
dc.date.issued	2017-12-22	en_US
dc.identifier.citation	Proceedings of the IEEE International Conference on Computer Vision, 2017, 2017-October pp. 3667 - 3676	en_US
dc.identifier.isbn	9781538610329	en_US
dc.identifier.issn	1550-5499	en_US
dc.identifier.uri	http://hdl.handle.net/10453/126805
dc.description.abstract	© 2017 IEEE. Existing video event classification approaches suffer from limited human-labeled semantic annotations. Weak semantic annotations can be harvested from Web-knowledge without involving any human interaction. However such weak annotations are noisy, thus can not be effectively utilized without distinguishing its reliability. In this paper, we propose a novel approach to automatically maximize the utility of weak semantic annotations (formalized as the semantic relevance of video shots to the target event) to facilitate video event classification. A novel attention model is designed to determine the attention scores of video shots, where the weak semantic relevance is considered as atten-tional guidance. Specifically, our model jointly optimizes two objectives at different levels. The first one is the classification loss corresponding to video-level groundtruth labels, and the second is the shot-level relevance loss corresponding to weak semantic relevance. We use a long short-term memory (LSTM) layer to capture the temporal information carried by the shots of a video. In each timestep, the LSTM employs the attention model to weight the current shot under the guidance of its weak semantic relevance to the event of interest. Thus, we can automatically exploit weak semantic relevance to assist video event classification. Extensive experiments have been conducted on three complex large-scale video event datasets i.e., MEDTest14, ActivityNet and FCVID. Our approach achieves the state-of-the-art classification performance on all three datasets. The significant performance improvement upon the conventional attention model also demonstrates the effectiveness of our model.	en_US
dc.relation.ispartof	Proceedings of the IEEE International Conference on Computer Vision	en_US
dc.relation.isbasedon	10.1109/ICCV.2017.394	en_US
dc.title	Leveraging Weak Semantic Relevance for Complex Video Event Classification	en_US
dc.type	Conference Proceeding
utslib.citation.volume	2017-October	en_US
utslib.for	0801 Artificial Intelligence and Image Processing	en_US
pubs.embargo.period	Not known	en_US
pubs.organisational-group	/University of Technology Sydney
pubs.organisational-group	/University of Technology Sydney/Faculty of Engineering and Information Technology
pubs.organisational-group	/University of Technology Sydney/Faculty of Engineering and Information Technology/School of Software
utslib.copyright.status	closed_access
pubs.publication-status	Published	en_US
pubs.volume	2017-October	en_US

Abstract:

© 2017 IEEE. Existing video event classification approaches suffer from limited human-labeled semantic annotations. Weak semantic annotations can be harvested from Web-knowledge without involving any human interaction. However such weak annotations are noisy, thus can not be effectively utilized without distinguishing its reliability. In this paper, we propose a novel approach to automatically maximize the utility of weak semantic annotations (formalized as the semantic relevance of video shots to the target event) to facilitate video event classification. A novel attention model is designed to determine the attention scores of video shots, where the weak semantic relevance is considered as atten-tional guidance. Specifically, our model jointly optimizes two objectives at different levels. The first one is the classification loss corresponding to video-level groundtruth labels, and the second is the shot-level relevance loss corresponding to weak semantic relevance. We use a long short-term memory (LSTM) layer to capture the temporal information carried by the shots of a video. In each timestep, the LSTM employs the attention model to weight the current shot under the guidance of its weak semantic relevance to the event of interest. Thus, we can automatically exploit weak semantic relevance to assist video event classification. Extensive experiments have been conducted on three complex large-scale video event datasets i.e., MEDTest14, ActivityNet and FCVID. Our approach achieves the state-of-the-art classification performance on all three datasets. The significant performance improvement upon the conventional attention model also demonstrates the effectiveness of our model.

Please use this identifier to cite or link to this item:

http://hdl.handle.net/10453/126805