Towards Scalable One-Pass Analytics Using MapReduce

Towards Scalable One-Pass Analytics Using MapReduce,10.1109/IPDPS.2011.251,Edward Mazur,Boduo Li,Yanlei Diao,Prashant Shenoy

Towards Scalable One-Pass Analytics Using MapReduce   (Citations: 1)
BibTex | RIS | RefWorks Download
An integral part of many data-intensive applica- tions is the need to collect and analyze enormous datasets efficiently. Concurrent with such application needs is the in- creasing adoption of MapReduce as a programming model for processing large datasets using a cluster of machines. Current MapReduce systems, however, require the data set to be loaded into the cluster before running analytical queries, and thereby incur high delays to start query processing. Furthermore, existing systems are geared towards batch processing. In this paper, we seek to answer a fundamental question: what architectural changes are necessary to bring the benefits of the MapReduce computation model to incremental, one- pass analytics, i.e., to support stream processing and online aggregation? To answer this question, we first conduct a detailed empirical performance study of current MapReduce implementations including Hadoop and MapReduce Online using a variety of workloads. By doing so, we identify several drawbacks of existing systems for one-pass analytics. Based on the insights from our study, we list key design requirements for incremental one-pass analytics and argue for architectural changes of MapReduce systems to overcome their current limitations. We conclude by sketching an initial design of our new MapReduce-based platform for incremental one-pass analytics and showing promising preliminary results.
Published in 2011.
Cumulative Annual
View Publication
The following links allow you to view full publications. These links are maintained by other sources not affiliated with Microsoft Academic Search.
Sort by: