Gloss-Free End-to-End Sign Language Translation

Lin, K; Wang, X; Zhu, L; Sun, K; Zhang, B; Yang, Y

Gloss-Free End-to-End Sign Language Translation

Lin, K Wang, X Zhu, L Sun, K Zhang, B Yang, Y

Permalink

Publication Type:: Conference Proceeding
Citation:: Proceedings of the Annual Meeting of the Association for Computational Linguistics, 2023, 1, pp. 12904-12916
Issue Date:: 2023-01-01

Closed Access

	Filename	Description	Size
	2305.12876.pdf	Accepted version	3.92 MB	Adobe PDF	View/Open

Copyright Clearance Process

Recently Added
In Progress
Closed Access

This item is closed access and not available.

Full metadata record

Field	Value	Language
dc.contributor.author	Lin, K
dc.contributor.author	Wang, X
dc.contributor.author	Zhu, L
dc.contributor.author	Sun, K
dc.contributor.author	Zhang, B
dc.contributor.author	Yang, Y https://orcid.org/0000-0002-0512-880X
dc.date.accessioned	2024-03-18T00:49:23Z
dc.date.available	2024-03-18T00:49:23Z
dc.date.issued	2023-01-01
dc.identifier.citation	Proceedings of the Annual Meeting of the Association for Computational Linguistics, 2023, 1, pp. 12904-12916
dc.identifier.isbn	9781959429722
dc.identifier.issn	0736-587X
dc.identifier.uri	http://hdl.handle.net/10453/176830
dc.description.abstract	In this paper, we tackle the problem of sign language translation (SLT) without gloss annotations. Although intermediate representation like gloss has been proven effective, gloss annotations are hard to acquire, especially in large quantities. This limits the domain coverage of translation datasets, thus handicapping real-world applications. To mitigate this problem, we design the Gloss-Free End-to-end sign language translation framework (GloFE). Our method improves the performance of SLT in the gloss-free setting by exploiting the shared underlying semantics of signs and the corresponding spoken translation. Common concepts are extracted from the text and used as a weak form of intermediate representation. The global embedding of these concepts is used as a query for cross-attention to find the corresponding information within the learned visual features. In a contrastive manner, we encourage the similarity of query results between samples containing such concepts and decrease those that do not. We obtained state-of-the-art results on large-scale datasets, including OpenASL and How2Sign.
dc.language	en
dc.relation.ispartof	Proceedings of the Annual Meeting of the Association for Computational Linguistics
dc.rights	info:eu-repo/semantics/closedAccess
dc.title	Gloss-Free End-to-End Sign Language Translation
dc.type	Conference Proceeding
utslib.citation.volume	1
pubs.organisational-group	University of Technology Sydney
pubs.organisational-group	University of Technology Sydney/Faculty of Engineering and Information Technology
utslib.copyright.status	closed_access	*
dc.date.updated	2024-03-18T00:49:20Z
pubs.publication-status	Published
pubs.volume	1

Abstract:

In this paper, we tackle the problem of sign language translation (SLT) without gloss annotations. Although intermediate representation like gloss has been proven effective, gloss annotations are hard to acquire, especially in large quantities. This limits the domain coverage of translation datasets, thus handicapping real-world applications. To mitigate this problem, we design the Gloss-Free End-to-end sign language translation framework (GloFE). Our method improves the performance of SLT in the gloss-free setting by exploiting the shared underlying semantics of signs and the corresponding spoken translation. Common concepts are extracted from the text and used as a weak form of intermediate representation. The global embedding of these concepts is used as a query for cross-attention to find the corresponding information within the learned visual features. In a contrastive manner, we encourage the similarity of query results between samples containing such concepts and decrease those that do not. We obtained state-of-the-art results on large-scale datasets, including OpenASL and How2Sign.

Please use this identifier to cite or link to this item:

http://hdl.handle.net/10453/176830