Scientific Programming

Scientific Programming / 2005 / Article
Special Issue

Dynamic Grids and Worldwide Computing

View this Special Issue

Open Access

Volume 13 |Article ID 962135 | https://doi.org/10.1155/2005/962135

Rob Pike, Sean Dorward, Robert Griesemer, Sean Quinlan, "Interpreting the Data: Parallel Analysis with Sawzall", Scientific Programming, vol. 13, Article ID 962135, 22 pages, 2005. https://doi.org/10.1155/2005/962135

Interpreting the Data: Parallel Analysis with Sawzall

Received30 Dec 2005
Accepted30 Dec 2005

Abstract

Very large data sets often have a flat but regular structure and span multiple disks and machines. Examples include telephone call records, network logs, and web document repositories. These large data sets are not amenable to study using traditional database techniques, if only because they can be too large to fit in a single relational database. On the other hand, many of the analyses done on them can be expressed using simple, easily distributed computations: filtering, aggregation, extraction of statistics, and so on. We present a system for automating such analyses. A filtering phase, in which a query is expressed using a new procedural programming language, emits data to an aggregation phase. Both phases are distributed over hundreds or even thousands of computers. The results are then collated and saved to a file. The design – including the separation into two phases, the form of the programming language, and the properties of the aggregators – exploits the parallelism inherent in having data and computation distributed across many machines.

Copyright © 2005 Hindawi Publishing Corporation. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.


More related articles

187 Views | 1057 Downloads | 241 Citations
 PDF Download Citation Citation
 Order printed copiesOrder

Related articles

We are committed to sharing findings related to COVID-19 as quickly as possible. We will be providing unlimited waivers of publication charges for accepted research articles as well as case reports and case series related to COVID-19. Review articles are excluded from this waiver policy. Sign up here as a reviewer to help fast-track new submissions.