This Article 
 Bibliographic References 
 Add to: 
Semantic Abstractions in the Multimedia Domain
January/February 2003 (vol. 15 no. 1)
pp. 136-160

Abstract—Information searching by exactly matching content is traditionally a strong point of machine searching; this is not, however, how human memory works and is rarely satisfactory for advanced retrieval tasks in any domain—multimedia in particular, where the presentational aspects can be equally important to the semantic information content. A combined abstraction of the conceptual and presentational characteristics of multimedia applications, leading on the one hand to their conceptual structure (with classic semantics of the real-world modeled by entities, relationships, and attributes) and on the other to the presentational structure (including media type, logical structure, temporal synchronization, spatial (on the screen) "synchronization" and interactive behavior) is developed in this paper. Multimedia applications are construed as consisting of "Presentational Units:" elementary (a media object with play duration and screen position), and composite (recursive structures of PUs in the temporal, spatial, and logical dimensions). The fundamental concept introduced is that of Semantic Multimedia Abstractions (SMA): qualitative abstract descriptions of multimedia applications in terms of their conceptual and presentational properties at an adjustable level of abstraction. SMAs, which could be viewed as metadata, form an abstract space to be queried. A detailed study of possible abstractions (from multimedia applications to SMAs and SMA-to-SMA), a definition and query language for Semantic Multimedia Abstractions (SMA-L) and the corresponding SMA model (equivalent to extended OMT), as well as an implementation of a system capable of wrapping the presentational structure of XML-based documents complete this work, whose contribution lays in the classically fruitful boundary between AI, software engineering, and database research.

[1] S. Adali, P. Bonatti, M. Sapino, and V.S. Subrahmanian, “A Multi-Similarity Algebra,” Proc. SIGMOD `98 Conf. Management of Data, pp. 402-413, 1998.
[2] W. Al-Khatib, Y.F. Day, A. Ghafoor, and P.B. Berra, “Semantic Modeling and Knowledge Representation in Multimedia Databases,” IEEE Trans. Knowledge and Data Eng., vol. 11, no. 1, pp. 64-80, Jan./Feb. 1999.
[3] J.F. Allen, “Maintaining Knowledge about Temporal Intervals,” Comm. ACM, vol. 26, no. 11, pp. 832–843, 1983.
[4] K. Autio, “Abstraction of Behavior and Structure in Model-Based Diagnosis,” Proc. DX-95 Sixth Int'l Workshop Principles of Diagnosis, Oct. 1995.
[5] R. Bergmann and W. Wilke, “Building and Refining Abstract Planning Cases by Change of Representation Language,” J. Artificial Intelligence Research 3, pp. 53-118, 1995.
[6] K. Bohm and T. Rakow, “Metadata for Multimedia Documents,” SIGMOD Record, special issue, Dec. 1994.
[7] S. Boll and W. Klas, “ZYX—A Semantic Model for Multimedia Ddocuments and Presentations,” Proc. Eighth IFIP Conf. Data Semantics (DS-8): Semantic Issues in Multimedia Systems, Jan. 1999.
[8] A. Borgida, J. Mylopoulos, and H. Wong, “Generalization/Specialization as a Basis for Software Specification,” On Conceptual Modelling, M. Brodie, J. Mylopoulos, and J. Schmidt, eds., pp. 87-117, Springer-Verlag, 1982.
[9] M. Brodie and D. Ridjanovic, “On the Design and Specification of Database Transactions,” On Conceptual Modelling, M. Brodie, J. Mylopoulos, and J. Schmidt, eds., pp. 277-306, Springer-Verlag, 1982.
[10] W.W. Chu, C.C. Hsu, A.F. Cardenas, and R.K. Taira, “Knowledge-Based Image Retrieval with Spatial and Temporal Constructs,” IEEE Trans. Knowledge and Data Eng., vol. 10, no. 6, Nov./Dec. 1998.
[11] V. Delis and T. Hadzilacos, “Binary String Relations: A Foundation for Spatio-Temporal Knowledge Representation,” Proc. ACM Conf. Information and Knowledge Management (CIKM), 1999.
[12] J.D. Dionisio and A.F. Cardenas, “A Unified Model for Representing Multimedia, Timeline and Simulation Data,” IEEE Trans. Knowledge and Data Eng., vol. 10, no. 5, Sept./Oct. 1998.
[13] J.M. Egenhofer and R.J. Herring, “Categorizing Binary Topological Relationships Between Regions, Lines and Points in Geographic Databases,” technical report, Dept. of Surveying Eng., Univ. of Maine, 1990.
[14] Extensible Markup Language (XML),W3C Recommendations XML 1.0, http://www.w3.orgXML/, Feb. 1998.
[15] R. Fikes and N.J. Nilson, “STRIPS: A New Approach to the Application Theorem Proving to Problem Solving,” Artificial Intelligence, vol. 2, pp. 189-208, 1971.
[16] D. Gardelis, T. Hadzilacos, P. Kourouniotis, M. Koutlis, E. Megalou, “Automating the Generation of Multimedia Titles,” Proc. 10th Int'l Conf. Advanced Science and Technology (ICAST '94), Mar. 1994.
[17] F. Giunchiglia and T. Walsh, “A Theory of Abstraction,” Artificial Intelligence, vol. 56, nos. 2-3, pp. 323-390, 1992.
[18] W. Grosky, F. Fotouli, and I. Sethi, “Using Metadata for Intelligent Browsing of Structured Media Objects,” SIGMOD Record, vol. 23, no. 4, Dec. 1994.
[19] N. Hirzalla, O. Megzari, and A. Karmouch, “An Object-Oriented Data Model and a Query Language for Multimedia Databases,” Proc. IEEE Int'l Conf. Electronics, Circuits, and Systems (ICECS '95), Dec. 1995.
[20] J.R. Hobbs, “Granularity,” Proc. Ninth Int'l Joint Conf. Artificial Intelligence (IJCAI), pp. 432-435, 1985.
[21] R.C. Holte, C. Drummond, M.B. Perez, R.M. Zimmer, and A.J. MacDonald, “Searching with Abstraction: A unifying Framework and a New High Performance Algorithm,” Proc. 10th Canadian Conf. Artifical Intelligence, pp. 263-270, 1994.
[22] R.C. Holte and T. Mkadmi, “Speeding up Problem Solving by Abstraction: A Graph Oriented Approach,” Artificial Intelligence, special issue on empirical artifical intelligence, P. Cohen and B. Porter, eds. Spring 1996.
[23] T. Imielinski, “Domain Abstraction and Limited Reasoning,” Proc. 10th Int'l Joint Conf. Artifical Intelligence, pp. 997-1003, 1987.
[24] ISO/IEC 10166-2:1991, Document Filing and Retrieval (DFR)—Part 2: Protocol Specification, 1991.
[25] Y. Iwasaki, “Reasoning with Multiple Abstraction Models,” Proc. Fourth Int'l Workshop Qualitative Physics, 1990.
[26] H. Jagadish, A. Medelzon, and T. Milo, “Similarity-Based Queries,” Proc. ACM Principles of Database Systems (PODS), pp. 36-45, May 1995.
[27] J.N. Johnson-Lairdm, Mental Models. Harvard Univ. Press. 1983.
[28] V. Kashyap, K. Shah, and A. Sheth, “Metadata for Building the MultiMedia Patch Quilt,” Multimedia Database Systems: Issues and Research Directions, S. Jajodia and V.S. Subrahmaniun, eds., Springer-Verlag, 1995.
[29] W. Klas, E.J. Neuhold, and M. Schrefl, “Using an Object-Oriented Approach to Model Multimedia Data,” Computer Comm. vol. 13, no. 4, pp. 204-216, May 1990.
[30] W. Klas and A. Sheth, “Metadata for Digital Media,” SIGMOD Record, special issue, Dec. 1994.
[31] H. Khalfallah and A. Karmouch, “An Architecture and a Data Model for Integrated Multimedia Documents and Presentational Applications,” ACM Multimedia Systems, vol. 3, pp. 238-250, 1995.
[32] C.A. Knoblock, “Automatically Generating Abstractions for Planning,” Artificial Intelligence, vol. 68, no. 2, 1994.
[33] A.Y. Levy, "Creating Abstractions Using Relevance Reasoning," Proc. AAAI-94, 1994.
[34] J.Z. Li, M.T. Özsu, D. Szafron, and V. Oria, “MOQL: A Multimedia Object Query Language,” Proc. Third Int'l Workshop Multimedia Information Systems, pp. 19-28, Sept. 1997.
[35] T.D.C. Little and A. Ghafoor, “Interval-Based Conceptual Models for Time-Dependent Multimedia Data,” IEEE Trans. Knowledge and Data Eng., vol. 5, no. 4, pp. 551-563, Aug. 1993.
[36] Macromedia Director, Macromedia Inc.,http:/, 1990.
[37] S. Marcus and V.S. Subrahmanian, “Foundations of Multimedia Database Systems,” J. ACM, vol. 43, no. 3, pp. 474-523, 1996.
[38] E. Megalou and T. Hadzilacos, “On Conceptual Modeling for Interactive Multimedia Presentations,” Proc. Second Int'l Conf. Multimedia Modeling '95 (MMM '95), p. 51, 1995.
[39] E. Megalou, T. Hadzilacos, and N. Mamoulis, “Conceptual Title Abstractions: Modeling and Querying Very Large Interactive Multimedia Repositories,” Proc. Third Int'l Conf. Multi-Media Modeling (MMM'96), 1996.
[40] C. Meghini, F. Rabitti, and C. Thanos, “Conceptual Document Modeling and Retrieval,” Computer Standards and Interfaces, vol. 11, pp.195-213, 1990/91.
[41] I. Mozetic, I. Bratko, and T. Urbancic, “Varying Levels of Abstraction in Qualitative Modelling,” Machine Intelligence, vol. 12, pp. 259-280, 1991.
[42] P. Nayak and A. Levy, “A Semantic Theory of Abstraction,” Proc. Int'l Joint Conf. Artificial Intelligence, 1995.
[43] S.B. Navathe, “Evolution of Data Modeling for Databases,” Comm. ACM, vol. 35, no. 9, Sept. 1992.
[44] R.J. Peters, A. Lipka, M.T. Oszu, and D. Szafron, “The Query Model and Query Language of TIGUKAT,” Technical Report 93-01, Dept. of Computer Science, Univ. of Alberta, Jan. 1995.
[45] D.A. Plaisted, “Theorem Proving with Abstraction,” Artificial Intelligence, vol. 16, pp. 47-108, 1981.
[46] A. Prieditis and B. Janakiraman, “Generating Effective Admissible Heuristics by Abstraction and Reconstitution,” Proc. Am. Assoc. Artificial Intelligence Conf., pp. 743-748, 1993.
[47] C.L. Chang, R.A. Stachowitz, and J.B. Combs, “Validation of Nonmonotonic Knowledge-Based Systems,” Proc. IEEE Int'l Conf. Tools for Artificial Intelligence, Nov. 1990.
[48] E.D. Sacerdoti, “Planning in a Hierarchy of Abstraction Spaces,” Artificial Intelligence, vol. 5, pp 115-135, 1974.
[49] G. Schloss and M. Wynblatt, “Providing Definition and Temporal Structure for Multimedia Data,” ACM Multimedia Systems, vol. 3, nos. 5/6, Nov. 1995.
[50] Standard Generalized Markup Language (SGML), International Standard ISO 8879. 1986.
[51] Synchronized Multimedia Integration Language (SMIL),W3C Recommendation SMIL 1.0,, 1998.
[52] G. Sygletos, “Abstracting XML-Based Documents to Semantic Multimedia Abstractions,” diploma thesis, Univ. of Patras, Greece, Dept. of Computer Eng. and Informatics (in Greek), 1999.
[53] N. Tryfona and T. Hadzilacos, “Geographic Applications Development: Models and Tools for the Conceptual Level,” Proc. Third ACM GIS Workshop (CIKM '95), Dec. 1995.
[54] ValMMeth, “Validation of a Multimedia Title Series Production Methodology,” Innovation Program IN34D, DGXIII, European Commission, CTI-R&D Unit 3, 1997-98.
[55] M. Vazirgiannis, Y. Theodoridis, and T. Sellis, “Spatio-Temporal Composition in Multimedia Applications,” Proc. IEEE Int'l Workshop Multimedia and Software Development, Software Eng., 1996.
[56] K. Wittenburg, “Introduction: Abstraction in Multimedia,” ACM Workshop Effective Abstractions in MMedia, 1995.

Index Terms:
Multimedia data model, semantic modeling, abstraction, semantic multimedia abstraction, spatio-temporal retrieval, multimedia query language, presentational unit.
Elina Megalou, Thanasis Hadzilacos, "Semantic Abstractions in the Multimedia Domain," IEEE Transactions on Knowledge and Data Engineering, vol. 15, no. 1, pp. 136-160, Jan.-Feb. 2003, doi:10.1109/TKDE.2003.1161587
Usage of this product signifies your acceptance of the Terms of Use.