Attend to Where and When: Cascaded Attention Network for Facial Expression Recognition

Qu, X; Zou, Z; Su, X; Zhou, P; Wei, W; Wen, S; Wu, D

Attend to Where and When: Cascaded Attention Network for Facial Expression Recognition

Qu, X Zou, Z Su, X Zhou, P Wei, W Wen, S

Wu, D

Permalink

Publisher:: Institute of Electrical and Electronics Engineers
Publication Type:: Journal Article
Citation:: IEEE Transactions on Emerging Topics in Computational Intelligence, 2022, 6, (3), pp. 580-592
Issue Date:: 2022-01-01

Closed Access

	Filename	Description	Size
	Attend to Where and When Cascaded Attention Network for Facial Expression Recognition.pdf	Published version	2.9 MB	Adobe PDF	View/Open

Copyright Clearance Process

Recently Added
In Progress
Closed Access

This item is closed access and not available.

Full metadata record

Field	Value	Language
dc.contributor.author	Qu, X
dc.contributor.author	Zou, Z
dc.contributor.author	Su, X
dc.contributor.author	Zhou, P
dc.contributor.author	Wei, W
dc.contributor.author	Wen, S https://orcid.org/0000-0001-8077-7001
dc.contributor.author	Wu, D
dc.date.accessioned	2023-02-28T04:26:06Z
dc.date.available	2023-02-28T04:26:06Z
dc.date.issued	2022-01-01
dc.identifier.citation	IEEE Transactions on Emerging Topics in Computational Intelligence, 2022, 6, (3), pp. 580-592
dc.identifier.issn	2471-285X
dc.identifier.issn	2471-285X
dc.identifier.uri	http://hdl.handle.net/10453/166558
dc.description.abstract	Recognizing human expression in videos is a challenging task due to dynamic changes in facial actions and diverse visual appearances. The key to design a reliable video-based expression recognition system is to extract robust spatial features and make full use of temporal modality characteristics. In this paper, we present a novel network architecture called Cascaded Attention Network (CAN) which is a cascaded spatiotemporal model incorporating with both spatial and temporal attention, tailored to video-level facial expression recognition. The cascaded fundamental model consists of a transfer convolutional network and Bidirectional Long Short-Term Memory (BiLSTM) network. Spatial attention is designed from the facial landmarks since facial expressions depend on the actions of key regions (eyebrows, eyes, nose, and mouth) on the face. Focusing on these key regions can help to decrease the effect of person-specific attributes. Meanwhile, the temporal attention is applied to automatically select the peak of expressions and aggregate the video-level representation. Our proposed CAN achieves the state-of-the-art performance on the three most widely used facial expression datasets: CK+ (99.03%), Oulu-CASIA (88.33%), and MMI (83.55%). Moreover, we conduct an extended experiment on a much more complex wild dataset AFEW and the experimental results further verify the generality of our attention mechanisms.
dc.language	en
dc.publisher	Institute of Electrical and Electronics Engineers
dc.relation.ispartof	IEEE Transactions on Emerging Topics in Computational Intelligence
dc.relation.isbasedon	10.1109/TETCI.2021.3070713
dc.rights	info:eu-repo/semantics/closedAccess
dc.title	Attend to Where and When: Cascaded Attention Network for Facial Expression Recognition
dc.type	Journal Article
utslib.citation.volume	6
pubs.organisational-group	/University of Technology Sydney
pubs.organisational-group	/University of Technology Sydney/Faculty of Engineering and Information Technology
pubs.organisational-group	/University of Technology Sydney/Strength - AAII - Australian Artificial Intelligence Institute
utslib.copyright.status	closed_access	*
pubs.consider-herdc	false
dc.date.updated	2023-02-28T04:26:03Z
pubs.issue	3
pubs.publication-status	Published
pubs.volume	6
utslib.citation.issue	3

Abstract:

Recognizing human expression in videos is a challenging task due to dynamic changes in facial actions and diverse visual appearances. The key to design a reliable video-based expression recognition system is to extract robust spatial features and make full use of temporal modality characteristics. In this paper, we present a novel network architecture called Cascaded Attention Network (CAN) which is a cascaded spatiotemporal model incorporating with both spatial and temporal attention, tailored to video-level facial expression recognition. The cascaded fundamental model consists of a transfer convolutional network and Bidirectional Long Short-Term Memory (BiLSTM) network. Spatial attention is designed from the facial landmarks since facial expressions depend on the actions of key regions (eyebrows, eyes, nose, and mouth) on the face. Focusing on these key regions can help to decrease the effect of person-specific attributes. Meanwhile, the temporal attention is applied to automatically select the peak of expressions and aggregate the video-level representation. Our proposed CAN achieves the state-of-the-art performance on the three most widely used facial expression datasets: CK+ (99.03%), Oulu-CASIA (88.33%), and MMI (83.55%). Moreover, we conduct an extended experiment on a much more complex wild dataset AFEW and the experimental results further verify the generality of our attention mechanisms.

Please use this identifier to cite or link to this item:

http://hdl.handle.net/10453/166558