Cross-Sentence Gloss Consistency for Continuous Sign Language Recognition

Rao, Q; Sun, K; Wang, X; Wang, Q; Zhang, B

Cross-Sentence Gloss Consistency for Continuous Sign Language Recognition

Rao, Q Sun, K Wang, X Wang, Q Zhang, B

Permalink

Publisher:: AAAI
Publication Type:: Conference Proceeding
Citation:: Proceedings of the AAAI Conference on Artificial Intelligence, 2024, 38, (5), pp. 4650-4658
Issue Date:: 2024-03-25

Open Access

Copyright Clearance Process

Recently Added
In Progress
Open Access

This item is open access.

Adobe PDF

Download Published versionAdobe PDF (516.45 kB)

View on publisher's site

View statistics

Full metadata record

Field	Value	Language
dc.contributor.author	Rao, Q
dc.contributor.author	Sun, K
dc.contributor.author	Wang, X
dc.contributor.author	Wang, Q
dc.contributor.author	Zhang, B
dc.date	2024-02-20
dc.date.accessioned	2025-03-10T01:23:09Z
dc.date.available	2025-03-10T01:23:09Z
dc.date.issued	2024-03-25
dc.identifier.citation	Proceedings of the AAAI Conference on Artificial Intelligence, 2024, 38, (5), pp. 4650-4658
dc.identifier.isbn	1-57735-887-2
dc.identifier.isbn	978-1-57735-887-9
dc.identifier.issn	2159-5399
dc.identifier.issn	2374-3468
dc.identifier.uri	http://hdl.handle.net/10453/185604
dc.description.abstract	Continuous sign language recognition (CSLR) aims to recognize gloss sequences from continuous sign videos. Recent works enhance the gloss representation consistency by mining correlations between visual and contextual modules within individual sentences. However, there still remain much richer correlations among glosses across different sentences. In this paper, we present a simple yet effective Cross-Sentence Gloss Consistency (CSGC), which enforces glosses belonging to a same category to be more consistent in representation than those belonging to different categories, across all training sentences. Specifically, in CSGC, a prototype is maintained for each gloss category and benefits the gloss discrimination in a contrastive way. Thanks to the welldistinguished gloss prototype, an auxiliary similarity classifier is devised to enhance the recognition clues, thus yielding more accurate results. Extensive experiments conducted on three CSLR datasets show that our proposed CSGC significantly boosts the performance of CSLR, surpassing existing state-of-the-art works by large margins (i.e., 1.6% on PHOENIX14, 2.4% on PHOENIX14-T, and 5.7% on CSLDaily).
dc.language	en
dc.publisher	AAAI
dc.relation.ispartof	Proceedings of the AAAI Conference on Artificial Intelligence
dc.relation.ispartof	AAAI Conference on Artificial Intelligence
dc.relation.isbasedon	10.1609/aaai.v38i5.28265
dc.rights	info:eu-repo/semantics/openAccess
dc.title	Cross-Sentence Gloss Consistency for Continuous Sign Language Recognition
dc.type	Conference Proceeding
utslib.citation.volume	38
utslib.location.activity	Vancouver, Canada.
utslib.copyright.status	open_access	*
pubs.consider-herdc	false
dc.date.updated	2025-03-10T01:23:04Z
pubs.finish-date	2024-02-27
pubs.issue	5
pubs.place-of-publication	USA
pubs.publication-status	Published
pubs.start-date	2024-02-20
pubs.volume	38
utslib.citation.issue	5
dc.location	USA

Abstract:

Continuous sign language recognition (CSLR) aims to recognize gloss sequences from continuous sign videos. Recent works enhance the gloss representation consistency by mining correlations between visual and contextual modules within individual sentences. However, there still remain much richer correlations among glosses across different sentences. In this paper, we present a simple yet effective Cross-Sentence Gloss Consistency (CSGC), which enforces glosses belonging to a same category to be more consistent in representation than those belonging to different categories, across all training sentences. Specifically, in CSGC, a prototype is maintained for each gloss category and benefits the gloss discrimination in a contrastive way. Thanks to the welldistinguished gloss prototype, an auxiliary similarity classifier is devised to enhance the recognition clues, thus yielding more accurate results. Extensive experiments conducted on three CSLR datasets show that our proposed CSGC significantly boosts the performance of CSLR, surpassing existing state-of-the-art works by large margins (i.e., 1.6% on PHOENIX14, 2.4% on PHOENIX14-T, and 5.7% on CSLDaily).

Please use this identifier to cite or link to this item:

http://hdl.handle.net/10453/185604