This Article 
 Bibliographic References 
 Add to: 
A Database Approach for Modeling and Querying Video Data
September/October 2000 (vol. 12 no. 5)
pp. 729-750

Abstract—Indexing video data is essential for providing content-based access. In this paper, we consider how database technology can offer an integrated framework for modeling and querying video data. As many concerns in video (e.g., modeling and querying) are also found in databases, databases provide an interesting angle to attack many of the problems. From a video applications perspective, database systems provide a nice basis for future video systems. More generally, database research will provide solutions to many video issues, even if these are partial or fragmented. From a database perspective, video applications provide beautiful challenges. Next generation database systems will need to provide support for multimedia data (e.g., image, video, audio). These data types require new techniques for their management (i.e., storing, modeling, querying, etc.). Hence, new solutions are significant. This paper develops a data model and a rule-based query language for video content-based indexing and retrieval. The data model is designed around the object and constraint paradigms. A video sequence is split into a set of fragments. Each fragment can be analyzed to extract the information (symbolic descriptions) of interest that can be put into a database. This database can then be searched to find information of interest. Two types of information are considered: 1) the entities (objects) of interest in the domain of a video sequence, and 2) video frames which contain these entities. To represent this information, our data model allows facts as well as objects and constraints. The model consists of two layers: 1) Feature & Content Layer (or Audiovisual Layer), intended to contain video visual features such as colors, contours, etc., 2) Semantic Layer, which provides the (conceptual) content dimension of videos. We present a declarative, rule-based, constraint query language that can be used to infer relationships about information represented in the model. Queries can refer to the form dimension (i.e., information of the Feature & Content Layer), to the content dimension (i.e., information of the Semantic Layer), or to both. A program of the language is a rule-based system formalizing our knowledge of a video target application and it can also be considered as a (deductive) video database on its own right. The language has both a clear declarative and operational semantics.

[1] S. Abiteboul, “Querying Semi-Structured Data,” Proc. Sixth Int'l Conf. Data Base Theory, 1997.
[2] S. Adali, K.S. Candan, S.-S. Chen, K. Erol, and V.S. Subrahmanian, “Advanced Video Information Systems,” ACM Multimedia Systems J., vol. 4, pp. 172-186, 1996.
[3] T.G.A. Smith and G. Davenport, "The Stratification System: A Design Environment for Random Access Video," Proc. Workshop Networking and Operating System Support for Digital Audio and Video, 1992.
[4] T.G. Aguierre-Smith and N. C. Pincever, “Parsing Movies in Context,” Proc. Summer 1991 Usenix Conf., pp. 157–168, June 1991.
[5] G. Ahanger, D. Benson, and T. Little, “Video Query Formulation,” Proc. Int'l Soc. Optical Eng., Storage and Retrieval for Image and Video Database III (SPIE '95), W. Niblack and R.C. Jain, eds., pp. 280–291, Feb. 1995.
[6] A. Aiken and E.L. Wimmers, “Solving Systems of Set Constraints (extended abstract),” Proc. Seventh Ann. IEEE Symp. Logic in Computer Science, I.C.S. Press, ed., pp. 329–340, 1992.
[7] J.F. Allen, “Maintaining Knowledge about Temporal Intervals,” Comm. ACM, vol. 26, no. 11, pp. 832–843, 1983.
[8] K.R. Apt, Logic Programming, Handbook of Theoretical Computer Science. Elsevier, 1991.
[9] E. Ardizzone and M. La Casia, “Automatic Video Database Indexing and Retrieval,” Multimedia Tools and Applications, vol. 4, no. 1, pp. 29-56, 1997.
[10] E. Ardizzone and M.-S. Hacid, “A Knowledge Representation and Reasoning Support for Video Data,” Proc. 11th IEEE Int'l Conf. Tools with Artificial Intelligence (ICTAI '99), Nov. 1999.
[11] F. Baader and K.U. Schulz, “On the Combination of Symbolic Constraints, Solution Domains, and Constraint Solvers,” Proc. Int'l Conf. Principles and Practice of Constraint Programming (CP '95), vol. 976, pp. 380–397, 1995.
[12] A.D. Bimbo, Visual Information Retrieval. Morgan Kaufmann, 1999.
[13] A.D. Bimbo, M. Campanai, and P. Nesi, “A Three-Dimensional Iconic Environment for Image Database Querying,” IEEE Trans. Software Eng., vol. 19, no. 1, pp. 997–1,011, Jan. 1993.
[14] A. Brodsky and Y. Kornatzky, The LyriC Language: Constraining Objects Proc. ACM SIGMOD Int'l Conf. Management of Data, 1995.
[15] P. Buneman, S. Davidson, G. Hillebrand, and D. Suciu, “A Query Language and Optimization Techniques for Unstructured Data,” Proc. SIGMOD, pp. 505–516, 1996.
[16] N. Chang and K. Fu, “Picture Query Languages for Pictorial Database Systems,” Computer, vol. 14, no. 11, pp. 23–33, Nov. 1981.
[17] J. Chomicki and T. Imielinski, “Relational Specifications of Infinite Query Answers,” ACM Trans. Database Systems, vol. 18, no. 2, pp. 181–223, June 1993.
[18] T.-S. Chua and L.-Q. Ruan, "A Video Retrieval andSequencing System," ACM Trans. Information Systems, vol. 13, no. 4, 1995, pp. 373-407.
[19] A. Colmerauer, "An Introduction to Prolog III," Comm. ACM, Vol. 33, No. 7, July 1990, pp. 69-90.
[20] J.M. Corridoni and A.D. Bimbo, “Structured Digital Video Indexing,” Proc. 13th Int'l Conf. Pattern Recognition—ICPR '96, 1996.
[21] J.M. Corridoni, A.D. Bimbo, D. Lucarella, and H. Wenxue, “Multi-perspective Navigation of Movies,” J. Visual Languages and Computing, vol. 7, pp. 445–466, 1996.
[22] M. Davis, “Media Streams: An Iconic Visual Language for Video Annotation,” Proc. IEEE Symp. on Visual Languages, CS Press, Los Alamitos, Calif., 1993, pp. 196-202.
[23] C. Decleir, M.-S. Hacid, and J. Kouloumdjian, “A Database Approach for Modeling and Querying Video Data,” Proc. 15th Int'l Conf. Data Eng., (ICDE '99), Mar. 1999.
[24] N. Dimitrova, “The Myth of Semantic Video Retrieval,” ACM Computing Surveys, vol. 27, no. 4, pp. 584–586, Dec. 1995.
[25] A. Duda, R. Weiss, and D.K. Gifford, “Content-Based Access to Algebraic Video,” technical report, MIT Laboratory for Computer Science 1994.
[26] A.K. Elmagarmid, H. Jiang et al., , Video Database System: Issues, Products and Applications, Kluwer, 1997.
[27] R. Fagin, “Fuzzy Queries in Multimedia Database Systems,” Proc. 1998 ACM SIGACT-SIGMOD-SIGART Symp. Principles of Database Systems, 1998.
[28] M. Flickner, H. Sawhney, W. Niblack, J. Ashley, Q. Huang, B. Dom, M. Gorkani, J. Hafner, D. Lee, D. Petkovic, D. Steele, and P. Yanker, “Query by Image and Video Content: The QBIC System,” Intelligent Multimedia Information Retrieval, M.T. Maybury, ed., chapter 1, pp. 7–22, 1996.
[29] D. Le Gall, “MPEG: A Video Compression Standard for Multimedia Applications,” Comm. ACM, Apr. 1991.
[30] S. Gibbs, C. Breiteneder, and D. Tsichritzis, “Audio/Video Database: An Object-Oriented Approach,” Proc. Int'l Conf. Data Eng., pp. 381-390, 1993.
[31] S. Gibbs, C. Breiteneder, and D. Tsichritzis, “Data Modeling of Time-Based Media,” Proc. ACM SIGMOD Int'l Conf. Management of Data, pp. 91-102, June 1994.
[32] H.M. Gladney, N.J. Belkin, Z. Ahmed, E.A. Fox, R. Ashany, and M. Zemankova, “Digital Library: Gross Structure and Requirements (Report from a Workshop),” Technical Report RJ 9840, IBM Research Report, May 1994.
[33] S. Grumbach and J. Su, "Dense-Order Constraint Databases," Proc. 14th ACM SIGACT-SIGMOD-SIGART Symp. Principles of Database Systems, pp. 66-77, 1995.
[34] S. Grumbach, J. Su, and C. Tollu, “Linear Constraint Query Languages Expressive Power and Complexity,” Proc. Int'l Workshop Logic and Computational Complexity (LCC '94), D. Leivant, ed., pp. 426–446, Oct. 1994.
[35] B.J. Haan, P. Kahn, V.A. Riley, J.H. Coombs, and N.K. Meyrowitz, “IRIS Hypermedia Services,” Comm. ACM, vol. 35, no. 1, pp. 35–51, Jan. 1991.
[36] H. Hampapur, R. Jain, and T.E. Weymouth, “Feature Based Digital Video Indexing,” technical report, Univ. of Michigan, 1994.
[37] A.G. Hauptmann and M.A. Smith, “Text, Speech, and Vision for Video Segmentation: The Informedia Project,” technical report, Carnegie Mellon Univ., 1995.
[38] K. Hirata and T. Kato, “Query by Visual Example,” Advances in Database Technology EDBT '92, Third Int'l Conf. Extending Database Technology, 1992.
[39] R. Hjelsvold and R. Midstraum, “Modeling and Querying Video Data,” Proc. 20th Very Large Data Bases Conf., pp. 686-694, 1994.
[40] M. Höhfeld and G. Smolka, “Definite Relations over Constraint Languages,” LILOG Report 53, IWBS, IBM Deutschland, Postfach 80 08 80, 7000 Stuttgart 80, Germany, Oct. 1988.
[41] L. Huang, J.C.-M. Lee, Q. Li, and W. Xiong, “An Experimental Video Database Management System Based on Advanced Object-Oriented Techniques,” Proc. Int'l Soc. Optical Eng. Storage and Retrieval for Image and Video Database IV (SPIE '96), K. Sethi and R.C. Jain, ed., pp. 158–169, Feb. 1996.
[42] H.T. Jiang and A.K. Elmagarmid, “WVTDB—A Semantic Content-Based Video Database System on the World Wide Web,” IEEE Trans. Knowledge and Data Eng., vol. 10, no. 6, pp. 947-966, Nov./Dec. 1998.
[43] T. Joseph and A.F. Cardenas, "PICQUERY: A High Level Query Language for Pictorial Database Management," IEEE Trans. Software Engineering, vol. 14, no. 5, pp. 630-638, May 1988.
[44] P.C. Kanellakis, G.M. Kuper, and P.Z. Revesz, Constraint Query Languages J. Computer and System Sciences, vol. 51, no. 1, pp. 26-52, 1995.
[45] D.B. Kemp and P.J. Stuckey, “Analysis Based Constraint Query Optimization,” Proc. 10th Int'l Conf. Logic Programming, D.S. Warren, ed., pp. 666–682, June 1993.
[46] M. Kifer, G. Lausen, and J. Wu, "Logical Foundations of Object-Oriented and Frame-Based Languages," J. ACM, vol. 42, pp. 741-843, 1995.
[47] M. Kifer and J. Wu, "A Logic for Object-Oriented Logic Programming (Maier's O-Logic Revisited)," ACM Symp. Principles of Database Systems,Philadelphia, pp. 379-393, Mar. 1989.
[48] A.Y. Levy, A.O. Mendelzon, and Y. Sagiv, “Answering Queries Using Views,” Proc. ACM Symp. Principles of Database Systems, pp. 95-104, May 1995.
[49] J.Z. Li, M.T. Ozsu, and D. Szafron, “Modeling of Moving Objects in a Video Database,” Proc. IEEE Int'l Conf. Multimedia Computing and Systems, pp. 336–343, June 1997.
[50] W. Li, S. Gauch, J. Gauch, and K.M. Pua, “VISION: A Digital Video Library,” ACM Digital Libraries, 1996.
[51] T.D.C. Little, G. Ahanger, R.J. Folz, J.F. Gibbon, F.W. Reeve, D.H. Schelleng, and D. Venkatesh, “A Digital On-Demand Video Service Supporting Content-based Queries,” Proc. ACM Multimedia Conf., pp. 427-436, Aug. 1993.
[52] W.E. Mackay and G. Davenport, “Virtual Video Editing in Interactive Multimedia Applications,” Comm. ACM, vol. 32, no. 7, pp. 802–810, July 1989.
[53] S. Marcus and V.S. Subrahmanian, “Foundations of Multimedia Database Systems,” J. ACM, vol. 43, no. 3, pp. 474-523, 1996.
[54] G. Mecca and A.J. Bonner, “Sequences, Datalog and Transducers,” Proc. 1995 Symp. Principles of Database Systems (PODS '95), pp. 23–35, May 1995.
[55] C. Meghini, “Towards a Logical Reconstruction of Image Retrieval,” Proc. In't Soc. Optical Eng., Storage and Retrieval for Image and Video Database IV (SPIE '96), I.K. Sethi and R.C. Jain, eds., pp. 108–119, Feb. 1996.
[56] K.L. Myers, “Hybrid Reasoning Using Universal Attachment,” Artificial Intelligence, vol. 67, pp. 329–375, 1994.
[57] S. Nestorov, S. Abiteboul, and R. Motwani, "Extracting Schema from Semistructured Data," Proc. ACM Int'l Conf. Management of Data (SIGMOD), ACM Press, New York, 1998, pp. 295-306.
[58] W. Niblack, R. Barber, W. Equitz, M. Flickner, D. Petkovic, and P. Yanker, “The QBIC Project: Querying Images by Content Using Color, Texture, and Shape” IS&T/SPIE Symp. Electronic Imaging; Science and Technology, Feb. 1993.
[59] E. Oomoto, “Design and Implementation of a Video-Object Database System,” IEEE Trans. Knowledge and Data Eng., vol. 5, no. 4, pp. 629-643, Aug. 1993.
[60] M. Ortega, Y. Rui, K. Chakrabarti, S. Mehrotra, and T.S. Huang, “Supporting Similarity Queries in MARS,” Proc. Fifth ACM Int'l Multimedia Conf., pp. 403–413, Nov. 1997.
[61] J. Paredaens, P. Peelman, and L. Tanca, “G-Log: A Declarative Graphical Query Languages,” Proc. Second Int'l Conf. Deductive and Object-Oriented Databases (DOOD '91), C. Delobel, M. Kifer, and Y. Masunaga, eds., pp. 108–128, Dec. 1991.
[62] P. Revesz, "Datalog Queries of Set Constraint Databases," Proc. Fifth Int'l Conf. Database Theory, Lecture Notes in Computer Science 893, pp. 424-438, 1995.
[63] W.C. Rounds, “Set Values for Unification Based Grammar Formalisms and Logic Programming,” Research Report CSLI-88-129, Stanford Univ., 1988.
[64] L.A. Rowe, J.S. Boreczky, and C.A. Eads, “Indexes for User Access to Large Video Databases,” Proc. Storage and Retrieval of Image and Video Databases II, 1995.
[65] P.A. Sistla, O. Wolfson, S. Chamberlain, and S. Dao, Modeling and Querying Moving Objects Proc. Int'l Conf. Data Eng., pp. 422-432, 1997.
[66] D. Srivastava, R. Ramakrishnan, and P.Z. Revesz, “Constraint Objects,” Proc. Second Int'l Workshop Principles and Practice of Constraint Programming (PPCP '94), pp. 218–228, 1994.
[67] D. Swanberg, C.-F. Shu, and R. Jain, “Knowledge Guided Parsing in Video Databases,” Proc. Image and Video Processing Conf.; Symp. Electronic Imaging: Science&Technology, vol. 1,908, pp. 13–24, Feb. 1993.
[68] Special Issue in Video Information Systems, ACM Trans. Information Systems, vol. 13, no. 4, Oct. 1995.
[69] D. Toman, "Point vs. Interval-Based Query Languages for Temporal Databases," Proc. Fifth ACM SIGACT/MOD/ART Symp. Principles of Database Systems, pp. 58-67, 1996.
[70] Y. Tonomura, “Video Handling Based on Structured Information for Hypermedia Systems,” Proc. Int'l Conf. Multimedia Information Systems '91, pp. 333–344, 1991.
[71] R. Weiss, A. Duda, and D.K. Gifford, “Content-Based Access to Algebraic Video,” Proc. Int'l Conf. Multimedia Computing and Systems, pp. 140-151, May 1994.
[72] A. Yoshitaka, Y. Hosoda, M. Yoshimitsu, M. Hirakawa, and T. Ichikawa, “VIOLONE: Video Retrieval by Motion Example,” J. Visual Languages and Computing, vol. 7, pp. 423–443, 1996.
[73] A. Zhang and S. Multani, “Implementation of Video Presentation in Database Systems,” Proc. Int'l Soc. Optical Eng., Storage and Retrieval for Image and Video Database IV (SPIE '96), I.K. Sethi and R.C. Jain, eds., pp. 228–238, Feb. 1996.

Index Terms:
Content-based access of video, video database, video query, video indexing, video representation, rule-based query language, object-oriented modeling, constraint query language.
Mohand-Saïd Hacid, Cyril Decleir, Jacques Kouloumdjian, "A Database Approach for Modeling and Querying Video Data," IEEE Transactions on Knowledge and Data Engineering, vol. 12, no. 5, pp. 729-750, Sept.-Oct. 2000, doi:10.1109/69.877505
Usage of this product signifies your acceptance of the Terms of Use.