This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
Volcano— An Extensible and Parallel Query Evaluation System
February 1994 (vol. 6 no. 1)
pp. 120-135

To investigate the interactions of extensibility and parallelism in database query processing, we have developed a new dataflow query execution system called Volcano. The Volcano effort provides a rich environment for research and education in database systems design, heuristics for query optimization, parallel query execution, and resource allocation. Volcano uses a standard interface between algebra operators, allowing easy addition of new operators and operator implementations. Operations on individual items, e.g., predicates, are imported into the query processing operators using support functions. The semantics of support functions is not prescribed; any data type including complex objects and any operation can be realized. Thus, Volcano is extensible with new operators, algorithms, data types, and type-specific methods. Volcano includes two novel meta-operators. The choose-plan meta-operator supports dynamic query evaluation plans that allow delaying selected optimization decisions until run-time, e.g., for embedded queries with free variables. The exchange meta-operator supports intra-operator parallelism on partitioned datasets and both vertical and horizontal inter-operator parallelism, translating between demand-driven dataflow within processes and data-driven dataflow between processes. All operators, with the exception of the exchange operator, have been designed and implemented in a single-process environment, and parallelized using the exchange operator. Even operators not yet designed can be parallelized using this new operator if they use and provide the interator interface. Thus, the issues of data manipulation and parallelism have become orthogonal, making Volcano the first implemented query execution engine that effectively combines extensibility and parallelism.

[1] D. S. Batory, J. R. Barnett, J. F. Garza, K. P. Smith, K. Tsukuda, B. C. Twichell, and T. E. Wise, "GENESIS: An extensible database management system,"IEEE Trans. Software Eng., vol. 14, p. 1711, Nov. 1988.
[2] D. Bitton and D. J. Dewitt, "Duplicate record elimination in large data files,"ACM Trans. Database Syst., pp. 255-265, June 1983.
[3] D. Bittonet al., "Parallel algorithms for the execution of relational database operation,"ACM Trans. Database Syst., vol. 8, no. 3, pp. 324-353, Sept. 1983.
[4] K. Bratbergsengen, "Hashing methods and relational algebra operations," inProc. Conf. Very Large Data Bases(Singapore), Aug. 1984, pp. 323-333.
[5] M. J. Carey, D. J. Dewitt, J. E. Richardson, and E. J. Shekita, "Object and file management in the EXODUS extensible database system," inProc. Twelfth Int. Conf. Very Large Databases, pp. 91-100, 1986.
[6] M. Careyet al., "A data model and query language for EXODUS," inProc. ACM SIGMOD Int. Conf. Management of Data, pp. 413-423, June 1988.
[7] M. Carey,et al., "The EXODUS extensible DBMS proiect: An overview," inReadings in Object-Oriented Databases, S. Zdonik and D. Maier, Eds. Menlo Park, CA: Morgan-Kaufman, 1989.
[8] J. V. Carlis, "HAS: A relational algebra operator, or divide is not enough to conquer," inProc. IEEE Conf. Data Eng., Los Angeles, CA, Feb. 1986, p. 254.
[9] D. D. Chamberlinet al., "A history and evaluation of system R,"Commun. ACM, vol. 24, no. 10, pp. 632-646, Oct. 1981.
[10] H.-T. Chou, D. J. Dewitt, R. H. Katz, and A. C. Klug, "Design and implementation of the Wisconsin storage system,"Software Practice and Experience, vol. 15, no. 10, pp. 943-962, Oct. 1985.
[11] D. J. DeWitt, S. Ghandeharadizeh, D. Schneider, A. Bricker, H. I. Hsiao, and R. Rasmussen, "The Gamma database machine project,"IEEE Trans. Knowledge Data Eng., vol. 2, p. 44, Mar. 1990.
[12] S. Fushimi, M. Kitsuregawa, and H. Tanaka, "An overview of the system software of a parallel relational database machine GRACE," inProc. Conf. Very Large Data Bases(Kyoto, Japan), Aug. 1986, pp. 209-219.
[13] G. Graefe and D.J. Dewitt, "The Exodus Optimizer Generator,"Proc. ACM SIGMOD 1987 Int'l Conf. Management of Data, ACM, New York, 1987, pp. 160- 172.
[14] G. Graefe, "Rule-based query optimization in extensible database systems," Ph.D. dissertation, Univ. of Wisconsin, Madison, WI, Aug. 1987.
[15] G. Graefe and D. Maier, "Query optimization in object-oriented Database systems: A prospectus," inAdvances in Object-Oriented Database Systems, Lecture Notes in Computer Science, vol. 334, K. R. Dittrich, Ed. New York: Springer-Verlag, Sept. 1988, p. 358.
[16] G. Graefe, "Relational division: Four algorithms and their performance," inProc. IEEE Conf. Data Eng., Los Angeles, CA, Feb. 1989, p. 94.
[17] G. Graefe and K. Ward, "Dynamic query evaluation plans," inProc. ACM SIGMOD, Portland, OR, May 1989, pp. 358-366.
[18] G. Graefe and S. S. Thakkar, "Tuning a parallel database algorithm on a shared-memory multiprocessor,"Software-Practice and Experience, vol. 22, no. 7, July 1992, p. 485.
[19] G. Graefe and D. L. Davison, "Encapsulation of Parallelism and Architecture-Independence in Extensible Database Query Processing," to appear inIEEE Trans. on Softw. Eng., vol. 19, no. 8, August 1993.
[20] G. Graefe, "Encapsulation of parallelism in the Volcano query processing system," inProc. ACM SIGMOD Conf., Atlantic City, NJ, May 1990, p. 102.
[21] G. Graefe, "Parallel external sorting in Volcano," CU Boulder Comput. Sci. Tech. Rep. 459, Feb. 1990.
[22] H. Guttman, "R-trees: A dynamic index structure for spatial searching," inProc. ACM/SIGMOD Conf., 1984, pp. 47-56.
[23] L. Haas, W. Chang, G. Lohman, J. McPherson, P. F. Wilms, G. Lapis, B. Lindsay, H. Pirahesh, and M. J. Carey, and E. Shekita, "Starburst mid-flight: As the dust clears,"IEEE Trans. Knowledge Data Eng., vol. 2, p. 143, Mar. 1990.
[24] S. Hudson and R. King, "Cactis: A Self-Adaptive, Concurrent Implementation of an Object-Oriented Database Management System,"ACM Trans. Database Systems, Sept. 1989, pp. 291-321.
[25] T. Keller and G. Graefe, "The one-to-one match operator of the Volcano query processing system," Oregon Grad. Center, Comput. Sci. Tech. Rep., Beaverton, OR, June 1989.
[26] T. Keller, G. Graefe, and D. Maier, "Efficient assembly of complex objects," Proc. ACM SIGMOD Conf., Denver, CO. May 1991, p. 148.
[27] J. E. Richardson and M. J. Carey, "Programming constructs for database system implementations in EXODUS," in [51], pp. 208-219.
[28] J. E. Richardson, "E: A persistent systems implementation language," Comput. Sci. Tech. Rep. 868, Univ. Wisconson--Madison, Aug. 1989.
[29] K. Salem and H. Garcia-Molina, "Disk striping," inProc. IEEE Data Eng., pp. 336-342, Feb. 1986.
[30] H. J. Schek, H. B. Paul, M. H. Scholl, and G. Weikum, "The DASDBS project: Objectives, experiences, and future prospects,"IEEE Trans. Knowledge Data Eng., vol. 2, p. 25, Mar. 199.
[31] L. D. Shapiro, "Join processing in database systems with large main memories,"ACM Trans. Database Syst., vol. 11, no. 3, pp. 239-264, Sept. 1986.
[32] M. Stonebraker, "Retrospection on a database system,"ACM Trans. Database Syst., vol. 5, p. 225, June 1980.
[33] M. Stonebraker, P. Aoki, and M. Seltzer, "Parallelism in XPRS," UCB/Electronics Research Lab. Memo. M89/16, Berkeley, CA, Feb. 1989.
[34] M. Stonebraker, "The design of XPRS," inProc. 14th Int. Conf. VLDB, pp. 318-330, Los Angeles, Aug. 1988.
[35] M. Stonebraker, L. A. Rowe, and M. Hirohama, "The Implementation of Postgres,"IEEE Trans. Knowledge Data Eng., vol. 2, p. 125, Mar. 1990.

Index Terms:
query processing; parallel programming; resource allocation; data structures; Volcano; parallel query evaluation system; extensibility; database query processing; dataflow query execution system; query optimization; resource allocation; algebra operators; predicates; query processing operators; support functions; semantics; data type; type-specific methods.; novel meta-operators; choose-plan meta-operator; dynamic query evaluation plans; intra-operator parallelism; partitioned datasets; horizontal inter-operator parallelism; demand-driven dataflow; data-driven dataflow
Citation:
G. Graefe, "Volcano— An Extensible and Parallel Query Evaluation System," IEEE Transactions on Knowledge and Data Engineering, vol. 6, no. 1, pp. 120-135, Feb. 1994, doi:10.1109/69.273032
Usage of this product signifies your acceptance of the Terms of Use.