This Article 
 Bibliographic References 
 Add to: 
Encapsulation of Parallelism and Architecture-Independence in Extensible Database Query Execution
August 1993 (vol. 19 no. 8)
pp. 749-764

Emerging database application domains demand not only high functionality, but also high performance. To satisfy these two requirements, the Volcano query execution engine combines the efficient use of parallelism on a wide variety of computer architectures with an extensible set of query processing operators that can be nested into arbitrarily complex query evaluation plans. Volcano's novel exchange operator permits designing, developing, debugging, and tuning data manipulation operators in single-process environments but executing them in various forms of parallelism. The exchange operator shields the data manipulation operators from all parallelism issues. The design and implementation of the generalized exchange operator are examined. The authors justify their decision to support hierarchical architectures and argue that the exchange operator offers a significant advantage for development and maintenance of database query processing software. They discuss the integration of bit vector filtering into the exchange operator paradigm with only minor modifications.

[1] E. Babb, "Implementing a relational database by means of specialized hardware,"ACM TODS, vol. 4, no. 1, pp. 1-29, Mar. 79.
[2] D. S. Batory, J. R. Barnett, J. F. Garza, K. P. Smith, K. Tsukuda, B. C. Twichell, and T. E. Wise, "GENESIS: An extensible database management system,"IEEE Trans. Software Eng., vol. 14, no. 11, p. 1711, Nov. 1988.
[3] D. S. Batory, T. Y. Leung, and T. E. Wise, "Implementation concepts for an extensible data model and data language,"ACM TODS, vol. 13, no. 3, 1988.
[4] A. Bhide, "An analysis of three transaction processing architectures,"14th Int. Conf. Very-Large Databases, 1988, pp. 339-350.
[5] A. Bhide and M. Stonebraker, "A performance comparison of two architectures for fast transaction processing,"4th Int. Conf. on Data Eng., 1988, pp. 536-545.
[6] D. Bitton, D. J. DeWitt, and C. Turbyfill, "Benchmarking database systems--A systematic approach," inProc. 1983 Very Large Data-base conf., Oct. 1983.
[7] H. Boral, and D. J. DeWitt, "Database machines: An idea whose time has passed? A critique of the future of database machines," inDatabase Machines. Berlin, Germany: Springer-Verlag, 1983.
[8] H. Boral and D. DeWitt, "A methodology for database system performance evaluation," inProc. SIGMOD--Conf. Management of Data, Boston, MA, June 1984.
[9] H. Boral, "Parallelism in Bubba," inProc. Int. Symp. Databases in Parallel Distributed Syst., Austin, TX, Dec. 1988.
[10] H. Boral, W. Alexander, L. Clay, G. Copeland, S. Danforth, M. Franklin, B. Hart, M. Smith, and P. Valduriez, "Prototyping Bubba, a highly parallel database system,"IEEE Trans. Knowledge Data Eng., vol. 2, no. 1, p. 4, Mar. 1990.
[11] D. R. Cheriton, H. A. Goosen, and P. D. Boyle, "Paradigm: A highly scalable shared-memory multicomputer,"IEEE Comput., vol. 24, no. 2, p. 33, Feb. 1991.
[12] R. L. Cole and G. Graefe, "A dynamic plan optimizer," submitted for publication, Feb. 1993.
[13] G. Copeland, W. Alexander, E. Boughter, and T. Keller, "Data placement in bubba," inProc. ACM SIGMOD, Chicago, IL, June 1-3, 1988, pp. 99-109.
[14] W. Davison, "Parallel index building in Informix OnLine 6.0," inProc. ACM SIGMOD Conf., San Diego, CA, June 1992, p. 103.
[15] D. J. DeWitt and P. B. Hawthorn, "A performance evaluation of database machine architectures," inProc. Int. Conf. Very Large Data Bases, Cannes, France, Sept. 1981, p. 199.
[16] D. Dewitt, R. H. Gerber, G. Graefe, M. L. Heytens, K. B. Kumar, and M. Muralikrishna, "GAMMA--A high performance dataflow database machine," inProc. 12th Int. Conf. VLDB, Kyoto, Japan, Aug. 1986, pp. 228-237.
[17] D. DeWitt, S. Ghandeharizadeh, and D. Schneider, "A performance analysis of the gamma database machine," inProc. ACM-SIGMOD Int. Conf. Management Data, Chicago, IL, May 1988.
[18] D. J. DeWitt, S. Ghandeharizadeh, D. Schneider, A. Bricker, H. I. Hsiao, and R. Rasmussen, "The Gamma Database Machine Project,"IEEE Trans. Knowledge Data Eng., vol. 2, no. 1, p. 44, Mar. 1990.
[19] D. J. DeWitt, J. F. Naughton, and D. A. Schneider, "An evaluation of non-equijoin algorithms," inProc. 17th Int. Conf. Very Large Data Bases, Sept. 1991, pp. 443-452.
[20] D. J. DeWitt and J. Gray, "Parallel database systems: The future of high-performance database systems,"Commun. ACM, vol. 35, no. 6, p. 85, June 1992.
[21] S. Englert, J. Gray, R. Kocher, and P. Shah, "A benchmark of NonStop SQL Release 2 demonstrating near-linear speedup and scaleup on large databases," Tandem Computer Syst. Tech. Rep. 89.4, May 1989.
[22] M. F. Fang, R. C. T. Lee, and C. C. Chang, "The idea of de-clustering and its applications," inProc. 12th Int. Conf. VLDB, Kyoto, Japan, Aug. 1986, pp. 181-188.
[23] R. Gerber, "Dataflow query processing using multiprocessor hash-partitioned algorithms," Ph.D. dissertation, Comput. Sci. Tech. Rep. 672, Univ. of Wisconsin-Madison, Oct. 1986.
[24] S. Ghandeharizadeh and D. J. DeWitt, "Hybrid-range partitioning strategy: A new declustering strategy for multiprocessor database machines," inProc. Int. Conf. Very Large Data Bases, Brisbane, Australia, 1990, p. 481.
[25] J. R. Goodman and P. J. Woest, "The Wisconsin Multicube: A new large-scale cache-coherent multiprocessor," CS Tech. Rep. 766, Univ. of Wisconsin-Madison, Apr. 1988.
[26] G. Graefe, "Relational division: Four algorithms and their performance," inProc. IEEE Conf. Data Eng., Los Angeles, CA, Feb. 1989, p. 94.
[27] G. Graefe and K. Ward, "Dynamic query evaluation plans," inProc. ACM SIGMOD, Portland, OR, May 1989, pp. 358-366.
[28] G. Graefe, "Encapsulation of parallelism in the Volcano query processing system," inProc. ACM SIGMOD Conf., Atlantic City, NJ, May 1990, p. 102.
[29] G. Graefe and L. D. Shapiro, "Data compression and database performance," inProc. ACM/IEEE-CS Symp. Appl. Comput., Kansas City, MO, Apr. 1991.
[30] G. Graefe, R. L. Cole, D. L. Davison, W. J. McKenna, and R. H. Wolniewicz, "Extensible query optimization and parallel execution in Volcano," inQuery Processing for Advanced Database Applications, J. C. Freytag, G. Vossen, and D. Maier, Eds. San Mateo, CA: Morgan-Kaufman, 1993.
[31] G. Graefe and S. S. Thakkar, "Tuning a parallel database algorithm on a shared-memory multiprocessor,"Software-Practice and Experience, vol. 22, no. 7, July 1992, p. 485.
[32] G. Graefe and W. J. McKenna, "The Volcano Optimizer Generator: Extensibility and efficient search," inProc. IEEE Conf. Data Eng., Vienna, Austria, Apr. 1993, p. 209.
[33] G. Graefe, "Volcano, An extensible and parallel dataflow query processing system,"IEEE Trans. Knowledge Data Eng., vol. 5, no. 5, Oct. 1993.
[34] G. Graefe, "Query Evaluation Techniques for Large Databases,"ACM Computing Surveys, June 1993, pp. 73-171.
[35] G. Graefe, A. Linville, and L. D. Shapiro, "Sort versus hash revisited,"IEEE Trans. Knowledge Data Eng., to be published, 1993.
[36] J. Gray, "A census of Tandem system availability between 1985 and 1990," Tandem Computers Tech. Rep. 90.1, Jan. 1990.
[37] L. M. Haas, J. C. Freytag, G. M. Lohman, and H. Pirahesh, "Extensible query processing in Starburst," inProc. ACM SIGMOD.Portland, OR, May 1989, pp. 377-388.
[38] L. Haas, W. Chang, G. Lohman, J. McPherson, P. F. Wilms, G. Lapis, B. Lindsay, H. Pirahesh, M. J. Carey, and E. Shekita, "Starburst midflight: As the dust clears,"IEEE Trans. Knowledge Data Eng., vol. 2, no. 1, p. 143, Mar. 1990.
[39] W. Hong and M. Stonebraker, "Optimization of parallel query execution plans in XPRS," inProc. Int. Conf. Parallel and Distributed Inform. Syst., Miami Beach, FL, Dec. 1991.
[40] W. Hong, "Exploiting inter-operation parallelism in XPRS," inProc. ACM SIGMOD Conf., San Diego, CA, June 1992, p. 19.
[41] H. Hsiao and D. DeWitt, "Chained Declustering: A New Availability Strategy for Multiprocessor Database Machines,"Proc. Sixth Int'l Conf. Data Engineering, IEEE CS Press, Los Alamitos, Calif., Order No. 2025, 1990, pp. 456-465.
[42] K. A. Hua and C. Lee, "An adaptive data placement Scheme for parallel database computer systems," inProc. 16th Int. Conf. VLDB, Brisbane, Australia, Aug. 1990, pp. 493-506.
[43] T. Keller, G. Graefe, and D. Maier, "Efficient assembly of complex objects," Proc. ACM SIGMOD Conf., Denver, CO. May 1991, p. 148.
[44] S. Khoshafian and P. Valduriez, "Parallel execution strategies for declustered databases," inProc. 5th Int. Workshop Database Machines, Karuizawa, Japan, Oct. 1987.
[45] M. Kitsuregawa and Y. Ogawa, "Bucket spreading parallel hash: A new robust, parallel hash join method for data skew in the super database computer (SDC)," inProc. 16th Int. Conf. VLDB, Brisbane, Australia, Aug. 1990, pp. 210-221.
[46] D. Lenoski, J. Laudon, K. Gharachorloo, W. D. Weber, A. Gupta, J. Henessy, M. Horowitz, and M. S. Lam, "The Stanford Dash Multiprocessor,"IEEE Comput., vol. 25, no. 3, p. 63, Mar. 1992.
[47] B.G. Lindsay et al., "Computation and Communication in R*: A Distributed Data-base Manager,"ACM Trans. Computer Systems, Vol. 2, No. 1, Feb. 1984, pp. 24-28.
[48] P. M. Neches, "Hardware support for advanced data management systems,"IEEE Comput., vol. 17, no. 11, p. 29, Nov. 1984.
[49] P. M. Neches, "The Ynet: An interconnect structure for a highly concurrent data base computer system," inProc. 2nd Symp. Frontiers of Massively Parallel Computation, Fairfax, Oct. 1988.
[50] H. Pirahesh, C. Mohan, J. Cheng, T. S. Liu, and P. Selinger, "Parallelism in relational data base systems: architectural issues and design approaches,"Proc. 2nd Int. Symp. Databases in Parallel and Distrib. Syst., Dublin, July 1990.
[51] J. P. Richardson, H. Lu, and K. Mikkilineni, "Design and evaluation of parallel pipelined join algorithms," inProc. 1987 ACM SIGMOD Int. Conf. Management of Data, May 1987, pp. 399-409.
[52] J. E. Richardson and M. J. Carey, "Programming constructs for database system implementations in EXODUS," in [51], pp. 208-219.
[53] D. Schneider and D. Dewitt, "A performance evaluation of four parallel join algorithms in a shared-nothing multiprocessor environment," inProc. ACM SIGMOD Conf.(Portland, OR), May-June 1989, p. 110.
[54] L. D. Shapiro, S. Ni, and G. Graefe, "Full-time data compression: An ADT for database performance," submitted for publication, 1993.
[55] J. Shemer and P. M. Neches, "The genesis of a database computer,"IEEE Comput., vol. 17, no. 11, p. 42, Nov. 1984.
[56] J. W. Stamos and H. C. Young, "A symmetric fragment and replicate algorithm for distributed joins," IBM Tech. Rep. RJ7188, San Jose, CA, Dec. 5, 1989.
[57] M. Stonebraker, "The case for shared-nothing,"IEEE Database Eng., vol. 9, no. 1, Mar. 1986.
[58] M. Stonebraker, "The design of XPRS," inProc. 14th Int. Conf. VLDB, pp. 318-330, Los Angeles, Aug. 1988.
[59] Teradata,DBC/1012 Data Base Computer, Concepts and Facilities, Teradata Corp., Los Angeles, CA, 1983.
[60] R. H. Wolniewicz and G. Graefe, "Algebraic optimization of computations over scientific databases," inProc. Int. Conf. Very Large Data Bases, Dublin, Ireland, Aug. 1993, p. 13.
[61] E. Wong and R. H. Katz, "Distributing a database for parallelism," inProc. ACM SIGMOD Int. Conf: on Management of Data, San Jose, CA, pp. 23-29, May 1983.

Index Terms:
extensible database query execution; database application domains; high functionality; high performance; Volcano query execution engine; parallelism; computer architectures; query processing operators; arbitrarily complex query evaluation plans; debugging; data manipulation operators; exchange operator; generalized exchange operator; hierarchical architectures; database query processing software; bit vector filtering; distributed databases; parallel programming; query processing
G. Graefe, D.L. Davison, "Encapsulation of Parallelism and Architecture-Independence in Extensible Database Query Execution," IEEE Transactions on Software Engineering, vol. 19, no. 8, pp. 749-764, Aug. 1993, doi:10.1109/32.238579
Usage of this product signifies your acceptance of the Terms of Use.