SemGloVe: Semantic Co-Occurrences for GloVe from BERT

Gan, L; Teng, Z; Zhang, Y; Zhu, L; Wu, F; Yang, Y

SemGloVe: Semantic Co-Occurrences for GloVe from BERT

Gan, L Teng, Z Zhang, Y Zhu, L

Wu, F Yang, Y

Permalink

Publisher:: IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
Publication Type:: Journal Article
Citation:: IEEE/ACM Transactions on Audio Speech and Language Processing, 2022, 30, pp. 2696-2704
Issue Date:: 2022-01-01

Closed Access

	Filename	Description	Size
	SemGloVe_Semantic_Co-Occurrences_for_GloVe_From_BERT.pdf	Published version	1.01 MB	Adobe PDF	View/Open

Copyright Clearance Process

Recently Added
In Progress
Closed Access

This item is closed access and not available.

Full metadata record

Field	Value	Language
dc.contributor.author	Gan, L
dc.contributor.author	Teng, Z
dc.contributor.author	Zhang, Y
dc.contributor.author	Zhu, L https://orcid.org/0000-0002-4093-7557
dc.contributor.author	Wu, F
dc.contributor.author	Yang, Y https://orcid.org/0000-0002-0512-880X
dc.date.accessioned	2023-03-21T03:09:48Z
dc.date.available	2023-03-21T03:09:48Z
dc.date.issued	2022-01-01
dc.identifier.citation	IEEE/ACM Transactions on Audio Speech and Language Processing, 2022, 30, pp. 2696-2704
dc.identifier.issn	2329-9290
dc.identifier.issn	2329-9304
dc.identifier.uri	http://hdl.handle.net/10453/167927
dc.description.abstract	GloVe learns word embeddings by leveraging statistical information from word co-occurrence matrices. However, word pairs in the matrices are extracted from a predefined local context window, which might lead to limited word pairs and potentially semantic irrelevant word pairs. In this paper, we propose SemGloVe, which distills semantic co-occurrences from BERT into static GloVe word embeddings. Particularly, we propose two models to extract co-occurrence statistics based on either the masked language model or the multi-head attention weights of BERT. Our methods can extract word pairs limited by the local window assumption, and can define the co-occurrence weights by directly considering the semantic distance between word pairs. Experiments on several word similarity datasets and external tasks show that SemGloVe can outperform GloVe.
dc.language	English
dc.publisher	IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
dc.relation.ispartof	IEEE/ACM Transactions on Audio Speech and Language Processing
dc.relation.isbasedon	10.1109/TASLP.2022.3197316
dc.rights	info:eu-repo/semantics/closedAccess
dc.subject	0801 Artificial Intelligence and Image Processing, 0906 Electrical and Electronic Engineering
dc.subject.classification	Speech-Language Pathology & Audiology
dc.title	SemGloVe: Semantic Co-Occurrences for GloVe from BERT
dc.type	Journal Article
utslib.citation.volume	30
utslib.for	0801 Artificial Intelligence and Image Processing
utslib.for	0906 Electrical and Electronic Engineering
pubs.organisational-group	/University of Technology Sydney
pubs.organisational-group	/University of Technology Sydney/Faculty of Engineering and Information Technology
utslib.copyright.status	closed_access	*
dc.date.updated	2023-03-21T03:09:46Z
pubs.publication-status	Published
pubs.volume	30

Abstract:

GloVe learns word embeddings by leveraging statistical information from word co-occurrence matrices. However, word pairs in the matrices are extracted from a predefined local context window, which might lead to limited word pairs and potentially semantic irrelevant word pairs. In this paper, we propose SemGloVe, which distills semantic co-occurrences from BERT into static GloVe word embeddings. Particularly, we propose two models to extract co-occurrence statistics based on either the masked language model or the multi-head attention weights of BERT. Our methods can extract word pairs limited by the local window assumption, and can define the co-occurrence weights by directly considering the semantic distance between word pairs. Experiments on several word similarity datasets and external tasks show that SemGloVe can outperform GloVe.

Please use this identifier to cite or link to this item:

http://hdl.handle.net/10453/167927