Preemptive Hadoop jobs scheduling under a deadline

Liu, L; Zhou, Y; Liu, M; Xu, G; Chen, X; Fan, D; Wang, Q

Preemptive Hadoop jobs scheduling under a deadline

Liu, L Zhou, Y Liu, M

Xu, G

Chen, X Fan, D Wang, Q

Permalink

Publication Type:: Conference Proceeding
Citation:: Proceedings - 2012 8th International Conference on Semantics, Knowledge and Grids, SKG 2012, 2012, pp. 72 - 79
Issue Date:: 2012-12-01

Closed Access

	Filename	Description	Size
	2013002776OK.pdf	Published version	604.38 kB	Adobe PDF	View/Open

Copyright Clearance Process

Recently Added
In Progress
Closed Access

This item is closed access and not available.

Full metadata record

Field	Value	Language
dc.contributor.author	Liu, L	en_US
dc.contributor.author	Zhou, Y	en_US
dc.contributor.author	Liu, M https://orcid.org/0000-0003-4256-6531	en_US
dc.contributor.author	Xu, G https://orcid.org/0000-0003-4493-6663	en_US
dc.contributor.author	Chen, X	en_US
dc.contributor.author	Fan, D	en_US
dc.contributor.author	Wang, Q	en_US
dc.date.issued	2012-12-01	en_US
dc.identifier.citation	Proceedings - 2012 8th International Conference on Semantics, Knowledge and Grids, SKG 2012, 2012, pp. 72 - 79	en_US
dc.identifier.isbn	9780769547947	en_US
dc.identifier.uri	http://hdl.handle.net/10453/32923
dc.description.abstract	MapReduce has become the dominant programming model in a cloud-based data processing environment, such as Hadoop. First In First Out (FIFO) is the default job scheduling policy of Hadoop, but it cannot guarantee that the job will be completed by a specific deadline. Research has been focused on developing deadline-based MapReduce schedulers by using the non-preemptive scheduling approach. However, compared with the non-preemptive scheduling approach, the preemptive scheduling approach has some advantages, such as the total completion time and slot utilization. In this paper, we first formulated the preemptive scheduling problem under deadline constraint, and then we proposed preemptive scheduling algorithms. To our knowledge we implemented the first real preemptive job scheduler to meet deadlines on Hadoop. The experimental results indicate that the preemptive scheduling approach is promising, which is more efficient than the non-preemptive one for executing jobs under a certain deadline. © 2012 IEEE.	en_US
dc.relation.ispartof	Proceedings - 2012 8th International Conference on Semantics, Knowledge and Grids, SKG 2012	en_US
dc.relation.isbasedon	10.1109/SKG.2012.40	en_US
dc.title	Preemptive Hadoop jobs scheduling under a deadline	en_US
dc.type	Conference Proceeding
utslib.for	0806 Information Systems	en_US
dc.location.activity	Beijing, China
pubs.embargo.period	Not known	en_US
pubs.organisational-group	/University of Technology Sydney
pubs.organisational-group	/University of Technology Sydney/DVC (Teaching and Learning)
pubs.organisational-group	/University of Technology Sydney/DVC (Teaching and Learning)/Connected Intelligence Centre
pubs.organisational-group	/University of Technology Sydney/Faculty of Engineering and Information Technology
pubs.organisational-group	/University of Technology Sydney/Faculty of Engineering and Information Technology/School of Computer Science
pubs.organisational-group	/University of Technology Sydney/Strength - AAI - Advanced Analytics Institute Research Centre
utslib.copyright.status	closed_access
pubs.publication-status	Published	en_US

Abstract:

MapReduce has become the dominant programming model in a cloud-based data processing environment, such as Hadoop. First In First Out (FIFO) is the default job scheduling policy of Hadoop, but it cannot guarantee that the job will be completed by a specific deadline. Research has been focused on developing deadline-based MapReduce schedulers by using the non-preemptive scheduling approach. However, compared with the non-preemptive scheduling approach, the preemptive scheduling approach has some advantages, such as the total completion time and slot utilization. In this paper, we first formulated the preemptive scheduling problem under deadline constraint, and then we proposed preemptive scheduling algorithms. To our knowledge we implemented the first real preemptive job scheduler to meet deadlines on Hadoop. The experimental results indicate that the preemptive scheduling approach is promising, which is more efficient than the non-preemptive one for executing jobs under a certain deadline. © 2012 IEEE.

Please use this identifier to cite or link to this item:

http://hdl.handle.net/10453/32923