Interpreting the data: Parallel analysis with Sawzall

Interpreting the data: Parallel analysis with Sawzall,Scientific Programming,Rob Pike,Sean Dorward,Robert Griesemer,Sean Quinlan

Interpreting the data: Parallel analysis with Sawzall   (Citations: 163)
BibTex | RIS | RefWorks Download
Very large data sets often have a flat but regular structure and span multiple disks and machines. Examples include telephone call records, network logs, and web document reposi- tories. These large data sets are not amenable to study using traditional database techniques, if only because they can be too large tofit in a single relational database. On the other hand, many of the analyses done on them can be expressed using simple, easily distributed computations: filtering, aggregation, extraction of statistics, and so on. We present a system for automating such analyses. A filtering phase, in which a query is expressed using a new procedural programming language, emits data to an aggregation phase. Both phases are distributed over hundreds or even thousands of computers. The results are then collated and saved to a file. The design—including the separation into two phases, the form of the programming language, and the properties of the aggregators—exploits the parallelism inherent in having data and computation distributed across many machines.
Journal: Scientific Programming - SP , vol. 13, no. 4, pp. 277-298, 2005
Cumulative Annual
View Publication
The following links allow you to view full publications. These links are maintained by other sources not affiliated with Microsoft Academic Search.
Sort by: