On compressing deep models by low rank and sparse decomposition

Yu, X; Liu, T; Wang, X; Tao, D

On compressing deep models by low rank and sparse decomposition

Yu, X Liu, T Wang, X Tao, D

Permalink

Publication Type:: Conference Proceeding
Citation:: Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, 2017, 2017-January pp. 67 - 76
Issue Date:: 2017-11-06

Closed Access

	Filename	Description	Size
	On Compressing Deep Models by Low Rank and Sparse Decomposition.pdf	Published version	492.16 kB	Adobe PDF	View/Open

Copyright Clearance Process

Recently Added
In Progress
Closed Access

This item is closed access and not available.

Full metadata record

Field	Value	Language
dc.contributor.author	Yu, X	en_US
dc.contributor.author	Liu, T	en_US
dc.contributor.author	Wang, X	en_US
dc.contributor.author	Tao, D https://orcid.org/0000-0001-7225-5449	en_US
dc.date.issued	2017-11-06	en_US
dc.identifier.citation	Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, 2017, 2017-January pp. 67 - 76	en_US
dc.identifier.isbn	9781538604571	en_US
dc.identifier.uri	http://hdl.handle.net/10453/126448
dc.description.abstract	© 2017 IEEE. Deep compression refers to removing the redundancy of parameters and feature maps for deep learning models. Low-rank approximation and pruning for sparse structures play a vital role in many compression works. However, weight filters tend to be both low-rank and sparse. Neglecting either part of these structure information in previous methods results in iteratively retraining, compromising accuracy, and low compression rates. Here we propose a unified framework integrating the low-rank and sparse decomposition of weight matrices with the feature map reconstructions. Our model includes methods like pruning connections as special cases, and is optimized by a fast SVD-free algorithm. It has been theoretically proven that, with a small sample, due to its generalizability, our model can well reconstruct the feature maps on both training and test data, which results in less compromising accuracy prior to the subsequent retraining. With such a "warm start" to retrain, the compression method always possesses several merits: (a) higher compression rates, (b) little loss of accuracy, and (c) fewer rounds to compress deep models. The experimental results on several popular models such as AlexNet, VGG-16, and GoogLeNet show that our model can significantly reduce the parameters for both convolutional and fully-connected layers. As a result, our model reduces the size of VGG-16 by 15×, better than other recent compression methods that use a single strategy.	en_US
dc.relation.ispartof	Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017	en_US
dc.relation.isbasedon	10.1109/CVPR.2017.15	en_US
dc.title	On compressing deep models by low rank and sparse decomposition	en_US
dc.type	Conference Proceeding
utslib.citation.volume	2017-January	en_US
utslib.for	0806 Information Systems	en_US
pubs.embargo.period	Not known	en_US
pubs.organisational-group	/University of Technology Sydney
pubs.organisational-group	/University of Technology Sydney/Faculty of Engineering and Information Technology
utslib.copyright.status	closed_access
pubs.publication-status	Published	en_US
pubs.volume	2017-January	en_US

Abstract:

© 2017 IEEE. Deep compression refers to removing the redundancy of parameters and feature maps for deep learning models. Low-rank approximation and pruning for sparse structures play a vital role in many compression works. However, weight filters tend to be both low-rank and sparse. Neglecting either part of these structure information in previous methods results in iteratively retraining, compromising accuracy, and low compression rates. Here we propose a unified framework integrating the low-rank and sparse decomposition of weight matrices with the feature map reconstructions. Our model includes methods like pruning connections as special cases, and is optimized by a fast SVD-free algorithm. It has been theoretically proven that, with a small sample, due to its generalizability, our model can well reconstruct the feature maps on both training and test data, which results in less compromising accuracy prior to the subsequent retraining. With such a "warm start" to retrain, the compression method always possesses several merits: (a) higher compression rates, (b) little loss of accuracy, and (c) fewer rounds to compress deep models. The experimental results on several popular models such as AlexNet, VGG-16, and GoogLeNet show that our model can significantly reduce the parameters for both convolutional and fully-connected layers. As a result, our model reduces the size of VGG-16 by 15×, better than other recent compression methods that use a single strategy.

Please use this identifier to cite or link to this item:

http://hdl.handle.net/10453/126448