Adaptively attending to visual attributes and linguistic knowledge for captioning

Publication Type:
Conference Proceeding
MM 2017 - Proceedings of the 2017 ACM Multimedia Conference, 2017, pp. 1345 - 1353
Issue Date:
Filename Description Size
p1345-bin.pdfPublished version1.78 MB
Adobe PDF
Full metadata record
© 2017 Association for Computing Machinery. Visual content description has been attracting broad research attention in multimedia community because it deeply uncovers intrinsic semantic facet of visual data. Most existing approaches formulate visual captioning as machine translation task (i.e., from vision to language) via a top-down paradigm with global attention, which ignores to distinguish visual and non-visual parts during word generation. In this work, we propose a novel adaptive attention strategy for visual captioning, which can selectively attend to salient visual content based on linguistic knowledge. Specifically, we design a key control unit, termed visual gate, to adaptively decide "when" and "what" the language generator attend to during the word generation process. We map all the preceding outputs of language generator into a latent space to derive the representation of sentence structures, which assists the "visual gate" to choose appropriate attention timing. Meanwhile, we employ a bottom-up workflow to learn a pool of semantic attributes for serving as the propositional attention resources. We evaluate the proposed approach on two commonly-used benchmarks, i.e., MSCOCO and MSVD. The experimental results demonstrate the superiority of our proposed approach compared to several state-of-the-art methods.
Please use this identifier to cite or link to this item: