MapReduce optimization using regulated dynamic prioritization

MapReduce optimization using regulated dynamic prioritization,10.1145/1555349.1555384,Thomas Sandholm,Kevin Lai

MapReduce optimization using regulated dynamic prioritization   (Citations: 22)
BibTex | RIS | RefWorks Download
We present a system for allocating resources in shared data and compute clusters that improves MapReduce job scheduling in three ways. First, the system uses regulated and user-assigned priorities to offer different service levels to jobs and users over time. Sec- ond, the system dynamically adjusts resource allocations to fit the requirements of different job stages. Finally, the system automat- ically detects and eliminates bottlenecks within a job. We show experimentally using real applications that users can optimize not only job execution time but also the cost-benefit ratio or prioritiza- tion efficiency of a job using these three strategies. Our approach relies on a proportional share mechanism that continuously allo- cates virtual machine resources. Our experimental results show a 11−31% improvement in completion time and 4−187% improve- ment in prioritization efficiency for different classes of MapReduce jobs. We further show that delay intolerant users gain even more from our system.
Cumulative Annual
View Publication
The following links allow you to view full publications. These links are maintained by other sources not affiliated with Microsoft Academic Search.
Sort by: