Automatic Optimization for MapReduce Programs

Automatic Optimization for MapReduce Programs,Computing Research Repository,Eaman Jahani,Michael J. Cafarella,Christopher Ré

Automatic Optimization for MapReduce Programs   (Citations: 1)
BibTex | RIS | RefWorks Download
The MapReduce distributed programming framework has become popular, despite evidence that current implementations are inefficient, requiring far more hardware than a traditional relational databases to complete similar tasks. MapReduce jobs are amenable to many traditional database query optimizations (B+Trees for selections, column-store- style techniques for projections, etc), but existing systems do not apply them, substantially because free-form user code obscures the true data operation being performed. For example, a selection in SQL is easily detected, but a selection in a MapReduce program is embedded in Java code along with lots of other program logic. We could ask the programmer to provide explicit hints about the program's data semantics, but one of MapReduce's attractions is precisely that it does not ask the user for such information. This paper covers Manimal, which automatically analyzes MapReduce programs and applies appropriate data- aware optimizations, thereby requiring no additional help at all from the programmer. We show that Manimal successfully detects optimization opportunities across a range of data operations, and that it yields speedups of up to 1,121% on previously-written MapReduce programs.
Journal: Computing Research Repository - CORR , vol. abs/1104.3, 2011
Cumulative Annual
View Publication
The following links allow you to view full publications. These links are maintained by other sources not affiliated with Microsoft Academic Search.
    • ...Unfortunately, at best relational components are extracted from procedural code into a relational optimization framework [20], which is limited by the inability of the relational framework to match the expressiveness of procedural code.,Manimal [20] similarly extracts relational operations such as selection, projection, and data compression from user-defined functions through static data flow analysis...

    Zhenyu Guoet al. Spotting Code Optimizations in Data-Parallel Pipelines through PeriSCO...

Sort by: