LANA: A Language-Capable Navigator for Instruction Following and Generation

Wang, X; Wang, W; Shao, J; Yang, Y

LANA: A Language-Capable Navigator for Instruction Following and Generation

Wang, X Wang, W Shao, J Yang, Y

Permalink

Publisher:: Institute of Electrical and Electronics Engineers (IEEE)
Publication Type:: Conference Proceeding
Citation:: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2023, 2023-June, pp. 19048-19058
Issue Date:: 2023-01-01

Closed Access

	Filename	Description	Size
	LANA_A_Language-Capable_Navigator_for_Instruction_Following_and_Generation.pdf	Published version	1.85 MB	Adobe PDF	View/Open

Copyright Clearance Process

Recently Added
In Progress
Closed Access

This item is closed access and not available.

Full metadata record

Field	Value	Language
dc.contributor.author	Wang, X
dc.contributor.author	Wang, W
dc.contributor.author	Shao, J
dc.contributor.author	Yang, Y https://orcid.org/0000-0002-0512-880X
dc.date	2023-06-17
dc.date.accessioned	2024-03-18T01:01:20Z
dc.date.available	2024-03-18T01:01:20Z
dc.date.issued	2023-01-01
dc.identifier.citation	Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2023, 2023-June, pp. 19048-19058
dc.identifier.isbn	9798350301298
dc.identifier.issn	1063-6919
dc.identifier.uri	http://hdl.handle.net/10453/176832
dc.description.abstract	Recently, visual-language navigation (VLN) - entailing robot agents to follow navigation instructions - has shown great advance. However, existing literature put most emphasis on interpreting instructions into actions, only delivering 'dumb' wayfinding agents. In this article, we devise LANA, a language-capable navigation agent which is able to not only execute human-written navigation commands, but also provide route descriptions to humans. This is achieved by simultaneously learning instruction following and generation with only one single model. More specifically, two encoders, respectively for route and language encoding, are built and shared by two decoders, respectively for action prediction and instruction generation, so as to exploit cross-task knowledge and capture task-specific characteristics. Throughout pretraining and fine-tuning, both instruction following and generation are set as optimization objectives. We empirically verify that, compared with recent advanced task-specific solutions, LANA attains better performances on both instruction following and route description, with nearly half complexity. In addition, endowed with language generation capability, Lanacan explain to human its behaviours and assist human's wayfinding. This work is expected to foster future efforts towards building more trustworthy and socially-intelligent navigation robots.
dc.language	en
dc.publisher	Institute of Electrical and Electronics Engineers (IEEE)
dc.relation.ispartof	Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition
dc.relation.ispartof	2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
dc.relation.isbasedon	10.1109/CVPR52729.2023.01826
dc.rights	info:eu-repo/semantics/closedAccess
dc.title	LANA: A Language-Capable Navigator for Instruction Following and Generation
dc.type	Conference Proceeding
utslib.citation.volume	2023-June
pubs.organisational-group	University of Technology Sydney
pubs.organisational-group	University of Technology Sydney/Faculty of Engineering and Information Technology
utslib.copyright.status	closed_access	*
dc.date.updated	2024-03-18T01:01:18Z
pubs.finish-date	2023-06-24
pubs.publication-status	Published
pubs.start-date	2023-06-17
pubs.volume	2023-June

Abstract:

Recently, visual-language navigation (VLN) - entailing robot agents to follow navigation instructions - has shown great advance. However, existing literature put most emphasis on interpreting instructions into actions, only delivering 'dumb' wayfinding agents. In this article, we devise LANA, a language-capable navigation agent which is able to not only execute human-written navigation commands, but also provide route descriptions to humans. This is achieved by simultaneously learning instruction following and generation with only one single model. More specifically, two encoders, respectively for route and language encoding, are built and shared by two decoders, respectively for action prediction and instruction generation, so as to exploit cross-task knowledge and capture task-specific characteristics. Throughout pretraining and fine-tuning, both instruction following and generation are set as optimization objectives. We empirically verify that, compared with recent advanced task-specific solutions, LANA attains better performances on both instruction following and route description, with nearly half complexity. In addition, endowed with language generation capability, Lanacan explain to human its behaviours and assist human's wayfinding. This work is expected to foster future efforts towards building more trustworthy and socially-intelligent navigation robots.

Please use this identifier to cite or link to this item:

http://hdl.handle.net/10453/176832