Latent Memory-augmented Graph Transformer for Visual Storytelling

Qi, M; Qin, J; Huang, D; Shen, Z; Yang, Y; Luo, J

Latent Memory-augmented Graph Transformer for Visual Storytelling

Qi, M Qin, J Huang, D Shen, Z Yang, Y

Luo, J

Permalink

Publisher:: ACM
Publication Type:: Conference Proceeding
Citation:: MM 2021 - Proceedings of the 29th ACM International Conference on Multimedia, 2021, pp. 4892-4901
Issue Date:: 2021-10-17

Closed Access

	Filename	Description	Size
	3474085.3475236.pdf	Published version	1.61 MB	Adobe PDF	View/Open

Copyright Clearance Process

Recently Added
In Progress
Closed Access

This item is closed access and not available.

Full metadata record

Field	Value	Language
dc.contributor.author	Qi, M
dc.contributor.author	Qin, J
dc.contributor.author	Huang, D
dc.contributor.author	Shen, Z
dc.contributor.author	Yang, Y https://orcid.org/0000-0002-0512-880X
dc.contributor.author	Luo, J
dc.date.accessioned	2022-06-10T20:31:21Z
dc.date.available	2022-06-10T20:31:21Z
dc.date.issued	2021-10-17
dc.identifier.citation	MM 2021 - Proceedings of the 29th ACM International Conference on Multimedia, 2021, pp. 4892-4901
dc.identifier.isbn	9781450386517
dc.identifier.uri	http://hdl.handle.net/10453/158078
dc.description.abstract	Visual storytelling aims to automatically generate a human-like short story given an image stream. Most existing works utilize either scene-level or object-level representations, neglecting the interaction among objects in each image and the sequential dependency between consecutive images. In this paper, we present a novel Latent Memory-augmented Graph Transformer∼(LMGT ), a Transformer based framework for visual story generation. LMGT directly inherits the merits from the Transformer, which is further enhanced with two carefully designed components, i.e., a graph encoding module and a latent memory unit. Specifically, the graph encoding module exploits the semantic relationships among image regions and attentively aggregates critical visual features based on the parsed scene graphs. Furthermore, to better preserve inter-sentence coherence and topic consistency, we introduce an augmented latent memory unit that learns and records highly summarized latent information as the story line from the image stream and the sentence history. Experimental results on three widely-used datasets demonstrate the superior performance of LMGT over the state-of-the-art methods.
dc.language	en
dc.publisher	ACM
dc.relation.ispartof	MM 2021 - Proceedings of the 29th ACM International Conference on Multimedia
dc.relation.ispartof	Proceedings of the 29th ACM International Conference on Multimedia
dc.relation.isbasedon	10.1145/3474085.3475236
dc.rights	info:eu-repo/semantics/closedAccess
dc.title	Latent Memory-augmented Graph Transformer for Visual Storytelling
dc.type	Conference Proceeding
pubs.organisational-group	/University of Technology Sydney
pubs.organisational-group	/University of Technology Sydney/Faculty of Engineering and Information Technology
pubs.organisational-group	/University of Technology Sydney/Strength - AAII - Australian Artificial Intelligence Institute
utslib.copyright.status	closed_access	*
dc.date.updated	2022-06-10T20:31:19Z
pubs.publication-status	Published

Abstract:

Visual storytelling aims to automatically generate a human-like short story given an image stream. Most existing works utilize either scene-level or object-level representations, neglecting the interaction among objects in each image and the sequential dependency between consecutive images. In this paper, we present a novel Latent Memory-augmented Graph Transformer∼(LMGT ), a Transformer based framework for visual story generation. LMGT directly inherits the merits from the Transformer, which is further enhanced with two carefully designed components, i.e., a graph encoding module and a latent memory unit. Specifically, the graph encoding module exploits the semantic relationships among image regions and attentively aggregates critical visual features based on the parsed scene graphs. Furthermore, to better preserve inter-sentence coherence and topic consistency, we introduce an augmented latent memory unit that learns and records highly summarized latent information as the story line from the image stream and the sentence history. Experimental results on three widely-used datasets demonstrate the superior performance of LMGT over the state-of-the-art methods.

Please use this identifier to cite or link to this item:

http://hdl.handle.net/10453/158078