Locality-Sensitive State-Guided Experience Replay Optimization for Sparse-Reward in Online Recommendation

Chen, X; Yao, L; Mcauley, J; Guan, W; Chang, X; Wang, X

Locality-Sensitive State-Guided Experience Replay Optimization for Sparse-Reward in Online Recommendation

Chen, X Yao, L Mcauley, J Guan, W Chang, X

Wang, X

Permalink

Publisher:: ACM
Publication Type:: Conference Proceeding
Citation:: SIGIR 2022 - Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2022, pp. 1316-1325
Issue Date:: 2022-11-30

In Progress

	Filename	Description	Size
	Xiaocong_SIGIR2022_ER.pdf	Accepted version	1.59 MB	Adobe PDF	View/Open
	3477495.fm.pdf	Supporting information	5.22 MB	Adobe PDF	View/Open

Copyright Clearance Process

Recently Added
In Progress
Open Access

This item is being processed and is not currently available.

Full metadata record

Field	Value	Language
dc.contributor.author	Chen, X
dc.contributor.author	Yao, L
dc.contributor.author	Mcauley, J
dc.contributor.author	Guan, W
dc.contributor.author	Chang, X https://orcid.org/0000-0002-7778-8807
dc.contributor.author	Wang, X https://orcid.org/0000-0001-9582-3445
dc.date	2022-07-11
dc.date.accessioned	2022-10-07T00:32:10Z
dc.date.available	2022-04-01
dc.date.available	2022-10-07T00:32:10Z
dc.date.issued	2022-11-30
dc.identifier.citation	SIGIR 2022 - Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2022, pp. 1316-1325
dc.identifier.isbn	9781450387323
dc.identifier.uri	http://hdl.handle.net/10453/162373
dc.description.abstract	Online recommendation requires handling rapidly changing user preferences. Deep reinforcement learning (DRL) is an effective means of capturing users' dynamic interest during interactions with recommender systems. Generally, it is challenging to train a DRL agent in online recommender systems because of the sparse rewards caused by the large action space (e.g., candidate item space) and comparatively fewer user interactions. Leveraging experience replay (ER) has been extensively studied to conquer the issue of sparse rewards. However, they adapt poorly to the complex environment of online recommender systems and are inefficient in learning an optimal strategy from past experience. As a step to filling this gap, we propose a novel state-aware experience replay model, in which the agent selectively discovers the most relevant and salient experiences and is guided to find the optimal policy for online recommendations. In particular, a locality-sensitive hashing method is proposed to selectively retain the most meaningful experience at scale and a prioritized reward-driven strategy is designed to replay more valuable experiences with higher chance. We formally show that the proposed method guarantees the upper and lower bound on experience replay and optimizes the space complexity, as well as empirically demonstrate our model's superiority to several existing experience replay methods over three benchmark simulation platforms.
dc.language	en
dc.publisher	ACM
dc.relation	http://purl.org/au-research/grants/arc/DE180100251
dc.relation	http://purl.org/au-research/grants/arc/DP220103717
dc.relation.ispartof	SIGIR 2022 - Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval
dc.relation.ispartof	International ACM SIGIR Conference on Research and Development in Information Retrieval
dc.relation.isbasedon	10.1145/3477495.3532015
dc.rights	info:eu-repo/semantics/restrictedAccess
dc.title	Locality-Sensitive State-Guided Experience Replay Optimization for Sparse-Reward in Online Recommendation
dc.type	Conference Proceeding
utslib.location.activity	Madrid, Spain
pubs.organisational-group	/University of Technology Sydney
pubs.organisational-group	/University of Technology Sydney/Faculty of Engineering and Information Technology
pubs.organisational-group	/University of Technology Sydney/Strength - AAII - Australian Artificial Intelligence Institute
pubs.organisational-group	/University of Technology Sydney/Faculty of Engineering and Information Technology/School of Computer Science
utslib.copyright.status	in_progress	*
pubs.consider-herdc	false
dc.date.updated	2022-10-07T00:32:07Z
pubs.finish-date	2022-07-15
pubs.publication-status	Accepted
pubs.start-date	2022-07-11

Abstract:

Online recommendation requires handling rapidly changing user preferences. Deep reinforcement learning (DRL) is an effective means of capturing users' dynamic interest during interactions with recommender systems. Generally, it is challenging to train a DRL agent in online recommender systems because of the sparse rewards caused by the large action space (e.g., candidate item space) and comparatively fewer user interactions. Leveraging experience replay (ER) has been extensively studied to conquer the issue of sparse rewards. However, they adapt poorly to the complex environment of online recommender systems and are inefficient in learning an optimal strategy from past experience. As a step to filling this gap, we propose a novel state-aware experience replay model, in which the agent selectively discovers the most relevant and salient experiences and is guided to find the optimal policy for online recommendations. In particular, a locality-sensitive hashing method is proposed to selectively retain the most meaningful experience at scale and a prioritized reward-driven strategy is designed to replay more valuable experiences with higher chance. We formally show that the proposed method guarantees the upper and lower bound on experience replay and optimizes the space complexity, as well as empirically demonstrate our model's superiority to several existing experience replay methods over three benchmark simulation platforms.

Please use this identifier to cite or link to this item:

http://hdl.handle.net/10453/162373