2013 IEEE 29th International Conference on Data Engineering (ICDE) (2002)

San Jose, California

Feb. 26, 2002 to Mar. 1, 2002

ISBN: 0-7695-1531-2

pp: 0706

Per-Åke Larson , Microsoft

ABSTRACT

Partial preaggregation is a simple data reduction operator that can be applied to aggregation queries. Whenever we group and aggregate on a column set G, we can preaggregate on any column set that functionally determines G. Preaggregation can be used, for example, to reduce the input size to a join. Regular aggregation reduces the input to one record per group. Partial preaggregation exploit the fact that preaggregation need not be complete -if multiple record happen to be output for a group, they will be combined into the same group by the final aggregation. This paper describes a straightforward hash-based algorithm for partial preaggregation, discusses where it can be applied, and derives a mathematical model for estimating the output size. The effectiveness of the technique and the accuracy of the model are shown on both artificial and real data. It is also shown how to reduce memory requirement by combining partial preaggregation with the input phase of a subsequent join or sort operator. Partial preaggregation has been implemented, in part, in Microsoft SQL Server.

INDEX TERMS

CITATION

Per-Åke Larson,
"Data Reduction by Partial Preaggregation",

*2013 IEEE 29th International Conference on Data Engineering (ICDE)*, vol. 00, no. , pp. 0706, 2002, doi:10.1109/ICDE.2002.994787