Sim2RealVS: A New Benchmark for Video Stabilization with a Strong Baseline

Rao, Q; Yu, X; Navasardyan, S; Shi, H

Sim2RealVS: A New Benchmark for Video Stabilization with a Strong Baseline

Rao, Q Yu, X

Navasardyan, S Shi, H

Permalink

Publisher:: IEEE
Publication Type:: Conference Proceeding
Citation:: 2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2023, 00, pp. 5395-5404
Issue Date:: 2023-02-06

Closed Access

	Filename	Description	Size
	Sim2RealVS_ A New Benchmark for Video Stabilization with a Strong Baseline_Published.pdf	Published version	10.4 MB	Adobe PDF	View/Open

Copyright Clearance Process

Recently Added
In Progress
Closed Access

This item is closed access and not available.

Full metadata record

Field	Value	Language
dc.contributor.author	Rao, Q
dc.contributor.author	Yu, X https://orcid.org/0000-0002-0269-5649
dc.contributor.author	Navasardyan, S
dc.contributor.author	Shi, H
dc.date	2023-01-02
dc.date.accessioned	2024-03-10T23:04:04Z
dc.date.available	2024-03-10T23:04:04Z
dc.date.issued	2023-02-06
dc.identifier.citation	2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2023, 00, pp. 5395-5404
dc.identifier.isbn	9781665493468
dc.identifier.issn	2472-6737
dc.identifier.uri	http://hdl.handle.net/10453/176422
dc.description.abstract	Video stabilization is highly desirable when videos undergo severe jittering artifacts The difficulty of obtaining sufficient training data obstructs the development of video stabilization In this work we address this issue by presenting a Sim2RealVS benchmark with more than 1 300 pairs of shaky and stable videos Our benchmark is curated by an in game simulator with diverse scenes and various jittering effects Moreover we propose a simple yet strong baseline approach named Motion Trajectory Smoothing Network MTSNet by fully exploiting our Sim2RealVS data Our MTSNet consists of three main steps motion estimation global trajectory smoothing and frame warping In motion estimation we design a Motion Correction and Completion MCC module to rectify the optical flow with low confidence such as in textureless regions thus providing more consistent motion estimation for next steps Benefiting from our synthetic data we can explicitly learn a Trajectory Smoothing Transformer TST with ground truth supervision to smooth global trajectories In training TST we propose two fully supervised losses i e a motion magnitude similarity loss and a motion tendency similarity loss After training our TST is able to produce smooth motion trajectories for the shaky input videos Extensive qualitative and quantitative results demonstrate that our MTSNet achieves superior performance on both synthetic and real world data
dc.language	en
dc.publisher	IEEE
dc.relation	http://purl.org/au-research/grants/arc/DP220100800
dc.relation	http://purl.org/au-research/grants/arc/DE230100477
dc.relation.ispartof	2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)
dc.relation.ispartof	2023 IEEE/CVF Winter Conference on Applications of Computer Vision
dc.relation.ispartofseries	IEEE Winter Conference on Applications of Computer Vision
dc.relation.isbasedon	10.1109/wacv56688.2023.00537
dc.rights	info:eu-repo/semantics/closedAccess
dc.title	Sim2RealVS: A New Benchmark for Video Stabilization with a Strong Baseline
dc.type	Conference Proceeding
utslib.citation.volume	00
utslib.location.activity	Waikoloa, HI, USA
pubs.organisational-group	University of Technology Sydney
pubs.organisational-group	University of Technology Sydney/Faculty of Engineering and Information Technology
pubs.organisational-group	University of Technology Sydney/Strength - AAII - Australian Artificial Intelligence Institute
utslib.copyright.status	closed_access	*
dc.date.updated	2024-03-10T23:04:02Z
pubs.finish-date	2023-01-07
pubs.publication-status	Published
pubs.start-date	2023-01-02
pubs.volume	00

Abstract:

Video stabilization is highly desirable when videos undergo severe jittering artifacts The difficulty of obtaining sufficient training data obstructs the development of video stabilization In this work we address this issue by presenting a Sim2RealVS benchmark with more than 1 300 pairs of shaky and stable videos Our benchmark is curated by an in game simulator with diverse scenes and various jittering effects Moreover we propose a simple yet strong baseline approach named Motion Trajectory Smoothing Network MTSNet by fully exploiting our Sim2RealVS data Our MTSNet consists of three main steps motion estimation global trajectory smoothing and frame warping In motion estimation we design a Motion Correction and Completion MCC module to rectify the optical flow with low confidence such as in textureless regions thus providing more consistent motion estimation for next steps Benefiting from our synthetic data we can explicitly learn a Trajectory Smoothing Transformer TST with ground truth supervision to smooth global trajectories In training TST we propose two fully supervised losses i e a motion magnitude similarity loss and a motion tendency similarity loss After training our TST is able to produce smooth motion trajectories for the shaky input videos Extensive qualitative and quantitative results demonstrate that our MTSNet achieves superior performance on both synthetic and real world data

Please use this identifier to cite or link to this item:

http://hdl.handle.net/10453/176422