A Deadline-Aware Coflow Scheduling Approach for Big Data Applications

Tang, W; Wang, S; Li, D; Huang, T; Dou, W; Yu, S

A Deadline-Aware Coflow Scheduling Approach for Big Data Applications

Tang, W Wang, S Li, D Huang, T Dou, W Yu, S

Permalink

Publication Type:: Conference Proceeding
Citation:: IEEE International Conference on Communications, 2018, 2018-May
Issue Date:: 2018-07-27

Closed Access

	Filename	Description	Size
	08422563.pdf	Published version	375.7 kB	Adobe PDF	View/Open

Copyright Clearance Process

Recently Added
In Progress
Closed Access

This item is closed access and not available.

Full metadata record

Field	Value	Language
dc.contributor.author	Tang, W	en_US
dc.contributor.author	Wang, S	en_US
dc.contributor.author	Li, D	en_US
dc.contributor.author	Huang, T	en_US
dc.contributor.author	Dou, W	en_US
dc.contributor.author	Yu, S https://orcid.org/0000-0003-4485-6743	en_US
dc.date.issued	2018-07-27	en_US
dc.identifier.citation	IEEE International Conference on Communications, 2018, 2018-May	en_US
dc.identifier.isbn	9781538631805	en_US
dc.identifier.issn	1550-3607	en_US
dc.identifier.uri	http://hdl.handle.net/10453/133160
dc.description.abstract	© 2018 IEEE. Many datacenters usually process complex jobs such as MapReduce jobs. From a network perspective, most of these jobs trigger multiple parallel data flows, which comprise a coflow group semantically. When to schedule the jobs in datacenter or across multiple datacenters, most of current job schedulers have not considered the underlying network traffic load, which is suboptimal for jobs completion times. We present a new deadline-aware coflow scheduling approach called DCS, which takes the underlying network traffic load into consideration while guaranteeing high percentage of coflows that meet their deadlines. DCS aims to alleviate the network congestion in datacenters whose network worload are unbalanced, and it includes two stages for coflow scheduling: Firstly, it generates the task placement proposal by considering the underlying network workload. Secondly, it makes scheduling decision by estimating both task's execution time and transmission waiting time under the previous task placement proposal. The real-world data based simulation results have shown that DCS outperforms all existing solutions on reducing the percentage of coflows that miss their deadlines.	en_US
dc.relation.ispartof	IEEE International Conference on Communications	en_US
dc.relation.isbasedon	10.1109/ICC.2018.8422563	en_US
dc.title	A Deadline-Aware Coflow Scheduling Approach for Big Data Applications	en_US
dc.type	Conference Proceeding
utslib.citation.volume	2018-May	en_US
utslib.for	0801 Artificial Intelligence and Image Processing	en_US
pubs.embargo.period	Not known	en_US
pubs.organisational-group	/University of Technology Sydney
pubs.organisational-group	/University of Technology Sydney/Faculty of Engineering and Information Technology
utslib.copyright.status	closed_access
pubs.publication-status	Published	en_US
pubs.volume	2018-May	en_US

Abstract:

© 2018 IEEE. Many datacenters usually process complex jobs such as MapReduce jobs. From a network perspective, most of these jobs trigger multiple parallel data flows, which comprise a coflow group semantically. When to schedule the jobs in datacenter or across multiple datacenters, most of current job schedulers have not considered the underlying network traffic load, which is suboptimal for jobs completion times. We present a new deadline-aware coflow scheduling approach called DCS, which takes the underlying network traffic load into consideration while guaranteeing high percentage of coflows that meet their deadlines. DCS aims to alleviate the network congestion in datacenters whose network worload are unbalanced, and it includes two stages for coflow scheduling: Firstly, it generates the task placement proposal by considering the underlying network workload. Secondly, it makes scheduling decision by estimating both task's execution time and transmission waiting time under the previous task placement proposal. The real-world data based simulation results have shown that DCS outperforms all existing solutions on reducing the percentage of coflows that miss their deadlines.

Please use this identifier to cite or link to this item:

http://hdl.handle.net/10453/133160