SAANet: Siamese action-units attention network for improving dynamic facial expression recognition

Liu, D; Ouyang, X; Xu, S; Zhou, P; He, K; Wen, S

SAANet: Siamese action-units attention network for improving dynamic facial expression recognition

Liu, D Ouyang, X Xu, S Zhou, P He, K Wen, S

Permalink

Publisher:: Elsevier BV
Publication Type:: Journal Article
Citation:: Neurocomputing, 2020, 413, pp. 145-157
Issue Date:: 2020-11-06

Closed Access

	Filename	Description	Size
	1-s2.0-S092523122031050X-main.pdf		2.53 MB	Adobe PDF	View/Open

Copyright Clearance Process

Recently Added
In Progress
Closed Access

This item is closed access and not available.

Full metadata record

Field	Value	Language
dc.contributor.author	Liu, D
dc.contributor.author	Ouyang, X
dc.contributor.author	Xu, S
dc.contributor.author	Zhou, P
dc.contributor.author	He, K
dc.contributor.author	Wen, S https://orcid.org/0000-0002-5048-0319
dc.date.accessioned	2021-03-18T00:34:04Z
dc.date.available	2021-03-18T00:34:04Z
dc.date.issued	2020-11-06
dc.identifier.citation	Neurocomputing, 2020, 413, pp. 145-157
dc.identifier.issn	0925-2312
dc.identifier.issn	1872-8286
dc.identifier.uri	http://hdl.handle.net/10453/147306
dc.description.abstract	© 2020 Elsevier B.V. Facial expression recognition (FER) has a wide variety of applications ranging from human–computer interaction, robotics to health care. Although FER has made significant progress with the success of Convolutional Neural Network (CNN), it is still challenging especially for the video-based FER due to the dynamic changes in facial actions. Since the specific divergences exists among different expressions, we introduce a metric learning framework with a siamese cascaded structure that learns a fine-grained distinction for different expressions in video-based task. We also develop a pairwise sampling strategy for such metric learning framework. Furthermore, we propose a novel action-units attention mechanism tailored to FER task to extract spatial contexts from the emotion regions. This mechanism works as a sparse self-attention fashion to enable a single feature from any position to perceive features of the action-units (AUs) parts (eyebrows, eyes, nose, and mouth). Besides, an attentive pooling module is designed to select informative items over the video sequences by capturing the temporal importance. We conduct the experiments on four widely used datasets (CK+, Oulu-CASIA, MMI, and AffectNet), and also do experiment on the wild dataset AFEW to further investigate the robustness of our proposed method. Results demonstrate that our approach outperforms existing state-of-the-art methods. More in details, we give the ablation study of each component.
dc.language	en
dc.publisher	Elsevier BV
dc.relation.ispartof	Neurocomputing
dc.relation.isbasedon	10.1016/j.neucom.2020.06.062
dc.rights	info:eu-repo/semantics/closedAccess
dc.subject	08 Information and Computing Sciences, 09 Engineering, 17 Psychology and Cognitive Sciences
dc.subject.classification	Artificial Intelligence & Image Processing
dc.title	SAANet: Siamese action-units attention network for improving dynamic facial expression recognition
dc.type	Journal Article
utslib.citation.volume	413
utslib.for	08 Information and Computing Sciences
utslib.for	09 Engineering
utslib.for	17 Psychology and Cognitive Sciences
pubs.organisational-group	/University of Technology Sydney
pubs.organisational-group	/University of Technology Sydney/Faculty of Engineering and Information Technology
pubs.organisational-group	/University of Technology Sydney/Strength - AAII - Australian Artificial Intelligence Institute
utslib.copyright.status	closed_access	*
pubs.consider-herdc	false
dc.date.updated	2021-03-18T00:34:02Z
pubs.publication-status	Published
pubs.volume	413

Abstract:

© 2020 Elsevier B.V. Facial expression recognition (FER) has a wide variety of applications ranging from human–computer interaction, robotics to health care. Although FER has made significant progress with the success of Convolutional Neural Network (CNN), it is still challenging especially for the video-based FER due to the dynamic changes in facial actions. Since the specific divergences exists among different expressions, we introduce a metric learning framework with a siamese cascaded structure that learns a fine-grained distinction for different expressions in video-based task. We also develop a pairwise sampling strategy for such metric learning framework. Furthermore, we propose a novel action-units attention mechanism tailored to FER task to extract spatial contexts from the emotion regions. This mechanism works as a sparse self-attention fashion to enable a single feature from any position to perceive features of the action-units (AUs) parts (eyebrows, eyes, nose, and mouth). Besides, an attentive pooling module is designed to select informative items over the video sequences by capturing the temporal importance. We conduct the experiments on four widely used datasets (CK+, Oulu-CASIA, MMI, and AffectNet), and also do experiment on the wild dataset AFEW to further investigate the robustness of our proposed method. Results demonstrate that our approach outperforms existing state-of-the-art methods. More in details, we give the ablation study of each component.

Please use this identifier to cite or link to this item:

http://hdl.handle.net/10453/147306