Subscribe

Issue No.04 - April (2012 vol.23)

pp: 668-675

Paulo Sérgio Almeida , Universidade do Minho, Braga

Raquel Menezes , Universidade do Minho, Braga

Paulo Jesus , Universidade do Minho, Braga

DOI Bookmark: http://doi.ieeecomputersociety.org/10.1109/TPDS.2011.209

ABSTRACT

Aggregation of data values plays an important role on distributed computations, in particular, over peer-to-peer and sensor networks, as it can provide a summary of some global system property and direct the actions of self-adaptive distributed algorithms. Examples include using estimates of the network size to dimension distributed hash tables or estimates of the average system load to direct load balancing. Distributed aggregation using nonidempotent functions, like sums, is not trivial as it is not easy to prevent a given value from being accounted for multiple times; this is especially the case if no centralized algorithms or global identifiers can be used. This paper introduces Extrema Propagation, a probabilistic technique for distributed estimation of the sum of positive real numbers. The technique relies on the exchange of duplicate insensitive messages and can be applied in flood and/or epidemic settings, where multipath routing occurs; it is tolerant of message loss; it is fast, as the number of message exchange steps can be made just slightly above the theoretical minimum; and it is fully distributed, with no single point of failure and the result produced at every node.

INDEX TERMS

Aggregation, network size estimation, distributed sums, probabilistic estimation, self-configuration.

CITATION

Paulo Sérgio Almeida, Raquel Menezes, Paulo Jesus, "Extrema Propagation: Fast Distributed Estimation of Sums and Network Sizes",

*IEEE Transactions on Parallel & Distributed Systems*, vol.23, no. 4, pp. 668-675, April 2012, doi:10.1109/TPDS.2011.209REFERENCES

- [1] R. van Renesse, "The Importance of Aggregation,"
Proc. Future Directions in Distributed Computing, pp. 87-92, 2003.- [2] M. Jelasity, A. Montresor, and Ö. Babaoglu, "Gossip-Based Aggregation in Large Dynamic Networks,"
ACM Trans. Computer System, vol. 23, no. 3, pp. 219-252, 2005.- [3] D. Kempe, A. Dobra, and J. Gehrke, "Gossip-Based Computation of Aggregate Information,"
Proc. IEEE 44th Ann. Symp. Foundations of Computer Science (FOCS), pp. 482-491, 2003.- [4] I. Stoica, R. Morris, D.R. Karger, M.F. Kaashoek, and H. Balakrishnan, "Chord: A Scalable Peer-to-Peer Lookup Service for Internet Applications,"
Proc. SIGCOMM, pp. 149-160, 2001.- [5] I. Abraham and D. Malkhi, "Probabilistic Quorums for Dynamic Systems,"
Proc. 17th Int'l Symp. Distributed Computing, pp. 60-74, 2003.- [6] S. Madden, M.J. Franklin, J.M. Hellerstein, and W. Hong, "Tag: A Tiny Aggregation Service for Ad-Hoc Sensor Networks,"
Proc. Fifth Symp. Operating Systems Design and Implementation (OSDI), 2002.- [7] J. Li, K.R. Sollins, and D.-Y. Lim, "Implementing Aggregation and Broadcast over Distributed Hash Tables,"
Computer Comm. Rev., vol. 35, no. 1, pp. 81-92, 2004.- [8] S. Nath, P.B. Gibbons, S. Seshan, and Z.R. Anderson, "Synopsis Diffusion for Robust Aggregation in Sensor Networks,"
Proc. Second Int'l Conf. Embedded Networked Sensor Systems (SenSys), pp. 250-262, 2004.- [9] C. Baquero, P.S. Almeida, and R. Menezes, "Fast Estimation of Aggregates in Unstructured Networks,"
Proc. Fifth Int'l Conf. Autonomic and Autonomous Systems (ICAS), pp. 88-93, http://doi. ieeecomputersociety.org/10.1109 ICAS.2009.31, 2009.- [10] D. Psaltoulis, D. Kostoulas, I. Gupta, K. Birman, and A. Demers, "Practical Algorithms for Size Estimation in Large and Dynamic Groups," technical report, Univ. of Illinois, http://www.cs. cornell.edu/Info/Projects/ SpinglassPubs.html, 2004.
- [11] E.J. Gumbel,
Statistics of Extremes. Columbia Univ. Press, 1958.- [12] R.V. Hogg and A.F. Craig,
Introduction to Mathematical Statistics, fifth ed. Prentice-Hall, 1995.- [13] S. Madden, M. Franklin, J. Hellerstein, and W. Hong, "TAG: A Tiny AGgregation Service for Ad-Hoc Sensor Networks,"
ACM SIGOPS Operating Systems Rev., vol. 36, no. SI, pp. 131-146, Dec. 2002.- [14] S. Madden, R. Szewczyk, M. Franklin, and D. Culler, "Supporting Aggregate Queries over Ad-Hoc Wireless Sensor Networks,"
Proc. IEEE Fourth Workshop Mobile Computing Systems and Applications, pp. 49-58, Mar. 2002.- [15] Y. Birk, I. Keidar, L. Liss, A. Schuster, and R. Wolff, "Veracity Radius: Capturing the Locality of Distributed Computations,"
Proc. 25th Ann. ACM Symp. Principles of Distributed Computing (PODC), July 2006.- [16] Y. Birk, I. Keidar, L. Liss, and A. Schuster, "Efficient Dynamic Aggregation,"
Proc. 20th Int'l Symp. DIStributed Computing (DISC), pp. 90-104, Sept. 2006.- [17] K. Horowitz and D. Malkhi, "Estimating Network Size from Local Information,"
Information Processing Letters, vol. 88, no. 5, pp. 237-243, 2003.- [18] T. Shafaat, A. Ghodsi, and S. Haridi, "A Practical Approach to Network Size Estimation for Structured Overlays,"
Proc. Third Int'l Self-Organizing Systems, pp. 71-83, Dec. 2008.- [19] P. Jesus, C. Baquero, and P.S. Almeida, "Dependability in Aggregation by Averaging,"
Simpósio de Informatica (INForum), Sept. 2009.- [20] E.L. Merrer, A.-M. Kermarrec, and L. Massoulie, "Peer to Peer Size Estimation in Large and Dynamic Networks: A Comparative Study,"
Proc. IEEE 15th Int'l Symp. High Performance Distributed Computing, Jan. 2006.- [21] A. Ganesh, A. Kermarrec, E.L. Merrer, and L. Massoulié, "Peer Counting and Sampling in Overlay Networks Based on Random Walks,"
Distributed Computing, vol. 20, no. 4, pp. 267-278, 2007.- [22] L. Massoulié, E. Merrer, A.-M. Kermarrec, and A. Ganesh, "Peer Counting and Sampling in Overlay Networks: Random Walk Methods,"
Proc. 25th Ann. ACM Symp. Principles of Distributed Computing (PODC), 2006.- [23] S. Mane, S. Mopuru, K. Mehra, and J. Srivastava, "Network Size Estimation in a Peer-to-Peer Network," technical report, Dept. of Computer Science, Univ. of Minnesota, p. 12, Sept. 2005.
- [24] D. Kostoulas, D. Psaltoulis, I. Gupta, K. Birman, and A. Demers, "Decentralized Schemes for Size Estimation in Large and Dynamic Groups,"
Proc. IEEE Fourth Int'l Symp. Network Computing and Applications, pp. 41-48, 2005.- [25] D. Kostoulas, D. Psaltoulis, I. Gupta, K.P. Birman, and A.J. Demers, "Active and Passive Techniques for Group Size Estimation in Large-Scale and Dynamic Distributed Systems,"
J. Systems and Software, vol. 80, no. 10, pp. 1639-1658, Jan. 2007.- [26] J. Considine, F. Li, G. Kollios, and J.W. Byers, "Approximate Aggregation Techniques for Sensor Databases,"
Proc. 20th Int'l Conf. Data Eng. (ICDE), pp. 449-460, 2004.- [27] M. Bawa, H. Garcia-Molina, A. Gionis, and R. Motwani, "Estimating Aggregates on a Peer-To-Peer Network," Technical Report TR-2003-24, Stanford Univ., http://dbpubs.stanford.edu/pub2003-24, 2003.
- [28] A. Manjhi, S. Nath, and P. Gibbons, "Tributaries and Deltas: Efficient and Robust Aggregation in Sensor Network Streams,"
Proc. ACM SIGMOD Int'l Conf. Management of Data, pp. 287-298, 2005.- [29] P. Flajolet and G.N. Martin, "Probabilistic Counting Algorithms for Data Base Applications,"
J. Computer and System Sciences, vol. 31, no. 2, pp. 182-209, 1985.- [30] M. Durand and P. Flajolet, "Loglog Counting of Large Cardinalities (Extended Abstract),"
Proc. 11th Ann. European Symp. Algorithms, pp. 605-617, 2003.- [31] P. Flajolet, E. Fusy, O. Gandouet, and F. Meunier, "Hyperloglog: The Analysis of a Near-Optimal Cardinality Estimation Algorithm,"
Int'l Conf. Analysis of Algorithms (AofA), pp. 127-146, June 2007.- [32] D. Mosk-Aoyama and D. Shah, "Computing Separable Functions via Gossip,"
Proc. 25th Ann. ACM Symp. Principles of Distributed Computing, pp. 113-122, July 2006.- [33] E. Cohen, "Size-Estimation Framework with Applications to Transitive Closure and Reachability,"
J. Computer and System Sciences, vol. 55, no. 3, pp. 441-453, 1997.- [34] D. Kempe, A. Dobra, and J. Gehrke, "Gossip-Based Computation of Aggregate Information,"
Proc. IEEE 44th Ann. Symp. Foundations of Computer Science, pp. 482-491, 2003.- [35] M. Jelasity and A. Montresor, "Epidemic-Style Proactive Aggregation in Large Overlay Networks,"
Proc. 24th Int'l Conf. Distributed Computing Systems, pp. 102-109, Jan. 2004.- [36] F. Wuhib, M. Dam, R. Stadler, and A. Clemm, "Robust Monitoring of Network-Wide Aggregates through Gossiping,"
Proc. IFIP/IEEE 10th Int'l Symp. Integrated Network Management, pp. 226-235, May 2007.- [37] P. Jesus, C. Baquero, and P.S. Almeida, "Fault-Tolerant Aggregation by Flow Updating,"
Proc. Ninth IFIP Int'l Conf. Distributed Applications and Interoperable Systems (DAIS), pp. 73-86, 2009.- [38] P. Jesus, C. Baquero, and P.S. Almeida, "Fault-Tolerant Aggregation for Dynamic Networks,"
Proc. IEEE 29th Symp. Reliable Distributed Systems, pp. 37-43, 2010. |