This Article 
 Bibliographic References 
 Add to: 
Temporal Synchronization Models for Multimedia Data
July/August 1998 (vol. 10 no. 4)
pp. 612-631

Abstract—Multimedia information systems are considerably more complex than traditional ones in that they deal with very heterogeneous data such as text, video, and audio—characterized by different characteristics and requirements. One of the central characteristics of multimedia data is that of being heavily time-dependent in that they are usually related by temporal relationships that must be maintained during playout. In this paper, we discuss problems related to modeling temporal synchronization specifications for multimedia data. We investigate the characteristics that a model must possess to properly express the timing relationships among multimedia data, and we provide a classification for the various models proposed in the literature. For each devised category, several examples are presented, whereas the most representative models of each category are illustrated in detail. Then, the presented models are compared with respect to the devised requirements, and future research issues are discussed.

[1] J. Akkerhuis, A. Marks, J. Rosenberg, and M.S. Sherman, "Processable Multimedia Document Interchange Using ODA," Proc. EUUG Autumn Conf., pp. 167-177,Vienna, Sept. 1989.
[2] J.F. Allen, “Maintaining Knowledge about Temporal Intervals,” Comm. ACM, vol. 26, no. 11, pp. 832–843, 1983.
[3] A.F. Ates, M. Bilgic, S. Saito, and B. Sarikaya, "Using Timed CSP for Specification Verification and Simulation of Multimedia Synchronization," IEEE J. Selected Areas in Comm., vol. 14, no. 1, pp. 126-137, Jan. 1996.
[4] E. Bertino, B. Catania, E. Ferrari, and A. Trombetta, "Presentation Constraints for Multimedia Data," Proc. Second Int'l Workshop Multimedia Information Systems, pp. 53-57,West Point, N.Y., Sept. 1996.
[5] E. Bertino, E. Ferrari, and M. Stolf, "A System for the Specification and Generation of Multimedia Presentations," Proc. Third Int'l Workshop Multimedia Information Systems, pp. 83-91,Como, Italy, Sept. 1997.
[6] E. Bertino, E. Ferrari, and M. Stolf, "MPGS: An Interactive Tool for the Specification and Generation of Multimedia Presentations," technical report, Dept. of Computer Science, Univ. of Milan, Jan. 1998, submitted for publication.
[7] U. Bormann and C. Bormann, "ISOTEXT- A WYSIWYG Editing and Formatting System for ODA and SGML Documents," Offene Multifunktionale Arbeitsplaetze,Berlin, 1988.
[8] M. Buchanan and P. Zellweger, “Specifying Temporal Behavior In Hypermedia Documents” Proc. ACM Conf. on Hypertext, ACM Press, New York, 1992, pp. 262-271.
[9] M.C. Buchanan and P.T. Zellweger, "Automatically Generating Consistent Schedules for Multimedia Documents," Multimedia Systems J., vol. 1, no. 2, pp. 55-67, Sept. 1993.
[10] R.H. Campbell and N. Haberman, The Specification of Process Synchronization by Path Expressions, pp. 89–102, Springer-Verlag Dec. 1973.
[11] K.S. Candan, B. Prabhakaran, and V.S. Subrahmanian, “CHIMP : A Framework for Supporting Multimedia Document Authoring and Presentation,” Proc. ACM Multimedia Conf., Nov. 1996.
[12] S. Christodoulakis, M. Theodoridou, F. Ho, M. Papa, and A. Pathria, "Multimedia Document Presentation, Information Extraction, and Document Formation in MINOS: A Model and A System," ACM Trans. Office Information Systems, vol. 4, no. 4, pp. 345-383, Oct. 1986.
[13] W.W. Chu et al., “A Temporal Evolutionary Object-Oriented Data Model and Its Query Language for Medical Image Management,” Proc. Very Large Data Base Conf., 1992.
[14] J.E. Coolahan and N. Roussopoulos, "Timing Requirements for Time-Driven Systems Using Augmented Petri-Nets," IEEE Trans. Software Eng., vol. 9, pp. 603-616, Sept. 1983.
[15] T.H. Cormen,C.E. Leiserson, and R.L. Rivest,Introduction to Algorithms.Cambridge, Mass.: MIT Press/McGraw-Hill, 1990.
[16] J. Davies, D.M. Jackson, J.N. Reed, G.M. Reed, A.W. Roscoe, and S.A. Schneider, "Timed CSP: Theory and Practice," J.W. De Bakker, C. Huizing, W.P. De Roever, and G. Rosenberg, eds., Real-Time: Theory and Practice, Lecture Notes in Computer Science 600, pp. 640-675, Springer-Verlag, Berlin, 1991.
[17] Y.F. Day, S. Dagtas, M. Iino, A. Khokhar, and A. Ghafoor, “Object-Oriented Conceptual Modeling of Video Data,” Proc. Data Eng. (DE '95), pp. 401-408, 1995.
[18] W. Dreyer, A.K. Dittrich, and D. Schmidt, "An Object-Oriented Data Model for A Time Series Management Systems," Proc. Seventh Int'l Working Conf. Scientific and Statistical Database Management, pp. 186-195,Charlottesville, Va., Sept. 1994.
[19] R. Erfle, "Specification of Temporal Constraints in Multimedia Documents Using HyTime," Electronic Publishing, vol. 6, no. 4, pp. 397-411, Dec. 1993.
[20] R. Erfle, "HyTime as The Multimedia Document Model of Choice," Proc. IEEE Int'l Conf. Multimedia Computing and Systems, pp. 445-454,Boston, May 1994.
[21] M.L. Escobar-Molano and S. Ghandeharizadeh, "A Framework for Conceptualizing Structured Video," Proc. First Int'l Workshop Multimedia Information Systems, pp. 95-110,Arlington, Va., Sept. 1995.
[22] K. Fujikawa, S. Shimojo, T. Matsuura, S. Nishio, and H. Miyahara, "Multimedia Presentation System 'Harmony' with Temporal and Active Media," Proc. Usenix Conf., 1991.
[23] A. Ghafoor and P.B. Berra, "Multimedia Database Systems," Advanced Database Systems, Lecture Notes in Computer Science 759, pp. 397-411, Springer-Verlag, 1993.
[24] A. Ghafoor and Y.F. Day, "Object-Oriented Modeling and Querying of Multimedia Data," Proc. First Int'l Workshop Multimedia Information Systems, pp. 111-119,Arlington, Va., Sept. 1995.
[25] C. Ghezzi, D. Mandriolli, and A. Morzenti, "Trio: A Logic Language for Executable Specifications of Real-Time Systems," J. Systems and Software, vol. 12, no. 2, pp. 107-123, May 1990.
[26] S. Gibbs, "Composite Multimedia and Active Objects," Proc. Int'l Conf. Object-Oriented Programming: Systems, Languages, and Applications, Oct. 1991.
[27] S. Gibbs, C. Breiteneder, and D. Tsichritzis, “Audio/Video Database: An Object-Oriented Approach,” Proc. Int'l Conf. Data Eng., pp. 381-390, 1993.
[28] S. Gibbs, C. Breiteneder, and D. Tsichritzis, “Data Modeling of Time-Based Media,” Proc. ACM SIGMOD Int'l Conf. Management of Data, pp. 91-102, June 1994.
[29] S. Gibbs, D. Tsichritzis, A. Fitas, D. Konstantas, and Y. Yeorgaroudakis, "MUSE: A Multimedia Filing System," IEEE Software, vol. 4, no. 2, pp. 4-20, Mar. 1987.
[30] R.G. Herrtwich and L. Delgrossi, "ODA-Based Data Modeling in Multimedia Systems," Technical Report TR 90-043, Int'l Computer Science Inst., Berkeley, Calif., 1990.
[31] P. Hoepner, "Synchronizing the Presentation of Multimedia Objects—ODA Extensions," Proc. First Eurographics Workshop: Multimedia, Systems, Interaction, and Applications, pp. 87-100,Stockholm, Apr. 1991.
[32] H. Ishikawa et al., "The Design and Implementation of an Object-Oriented Multimedia Knowledge Base Management System," ACM Trans. Database Syst., vol. 18, no. 1, pp. 1-50, March 1993.
[33] A. Karmouch and J. Emery, "Playback Schedule Model for Multimedia Documents," IEEE MultiMedia, Vol. 3, No. 1, Spring 1996, pp. 50-61.
[34] W. Kim, E. Bertino, and J.F. Garza, “Composite Objects Revisited,” Proc. 1989 Int'l Conf. Management of Data, pp. 337-347, ACM SIGMOD, Portland, Ore., June 1989.
[35] L. Li, A. Karmouch, and N.D. Georganas, "Multimedia Teleorchestra with Independent Sources: Part 1—Temporal Modeling of Collaborative Multimedia Scenarios," ACM/Springer Multimedia Systems, vol. 1, no. 4, pp. 143-153, 1994.
[36] L. Li, A. Karmouch, and N.D. Georganas, "Multimedia Teleorchestra with Independent Sources: Part 2—Synchronization Algorithms," ACM/Springer Multimedia Systems, vol. 1, no. 4, pp. 154-165, 1994.
[37] T. Little and A. Ghafoor, "Synchronization and Storage for Multimedia Objects," IEEE J. Selected Areas in Comm., vol. 8, no. 3, pp. 413-427, Apr. 1990.
[38] T. Little and A. Ghafoor, "Network Considerations for Distributed Multimedia Object Composition and Communication," IEEE Network, vol. 4, no. 6, Nov . 1990, pp. 32-49.
[39] T.D.C. Little and A. Ghafoor, “Spatio-Temporal Composition of Distributed Multimedia Objects for Value Added Networks,” Computer, vol. 24, no. 10, pp. 42-50, Oct. 1991.
[40] T.D.C. Little and A. Ghafoor, “Interval-Based Conceptual Models for Time-Dependent Multimedia Data,” IEEE Trans. Knowledge and Data Eng., vol. 5, no. 4, pp. 551-563, Aug. 1993.
[41] S. Marcus and V.S. Subrahmanian, “Foundations of Multimedia Database Systems,” J. ACM, vol. 43, no. 3, pp. 474-523, 1996.
[42] Y. Masunaga, "Design Issues of OMEGA—An Object-Oriented Multimedia Database Management System," J. Information Processing, vol. 14, pp. 60-74, 1991.
[43] I. Meiri, "Combining Qualitative and Quantitative Constraints in Temporal Reasoning," Proc. AAAI, pp. 260-267, 1991.
[44] T. Meyer, W. Effelsberg, and R. Steinmetz, "A Taxonomy on Multimedia Synchronization," Proc. Fourth Int'l Workshop on Future Trends in Distributed Computing Systems, 1993.
[45] S.R. Newcomb, N.A. Kipp, and V.T. Newcomb, “HyTime—The Hypermedia/Time-Based Document Structuring Language,” Comm. ACM, vol. 34, no. 11, Nov. 1991.
[46] Office Document Architecture (ODA): An Interchange Format," no. 8613, ISO, 1986.
[47] E. Oomoto, “Design and Implementation of a Video-Object Database System,” IEEE Trans. Knowledge and Data Eng., vol. 5, no. 4, pp. 629-643, Aug. 1993.
[48] J.A. Orenstein and F.A. Manola, "PROBE Spatial Data Modeling and Query Processing in an Image Database Application," IEEE Trans. Software Engineering, vol. 14, no. 5, pp. 611-629, May 1988.
[49] M.J. Perez-Luque and T.D.C. Little, "A Temporal Reference Framework for Multimedia Synchronization," Proc. Int'l Workshop Multimedia Synchronization, May 1995.
[50] J.L. Peterson, “Petri Nets,” ACM Computing Surveys, vol. 9, no. 3, Sept. 1977.
[51] A. Pnueli, "Applications of Temporal Logic to the Specification and Verification of Concurrent Systems: A Survey of Current Trends," Lecture Notes in Computer Science 224, Springer-Verlag, 1986.
[52] A. Poggio, J. Garcia Luna Aceves, E. J. Craighill, D. Worthington, and J. Hight, "CCWS: A Computer-Based Multimedia Information System," Computer, vol. 18, no. 10, pp. 92-103, Oct. 1985.
[53] R.R. Razouk and C.V. Phelps, "Performance Analysis Using Timed Petri-Nets," Proc. Int'l Conf. Parallel Processing, pp. 126-129, Aug. 1984.
[54] Proc. Int'l Workshop on Research Issues in Data Eng.: Interoperability in Multidatabase Systems, IEEE CS Press, 1993.
[55] A. Rizk and L. Sauter, "Multicard: An Open Hypermedia System," Proc. ACM Conf. Hypertext,Milan, Italy, Nov. 1992.
[56] F. Sadri, "Three Recent Approaches to Temporal Reasoning," A. Galton, ed., Temporal Logics and Their Applications, pp. 121-168, 1987.
[57] A. Segev and R. Chandra, "A Data Model for Time-Series Analysis," Lecture Notes in Computer Science, no. 759, Springer Verlag, 1993.
[58] R. Steinmetz,"Synchronization Properties in Multimedia Systems," IEEE J. Selected Areas in Comm., vol. 8, no. 3, Apr. 1990, pp. 401-412.
[59] R. Steinmetz, "Multimedia Synchronization Techniques: Experience Based on Different System Structures," Proc. IEEE Multimedia, CS Press, Los Alamitos, Calif., 1992, pp. 306-314.
[60] Multimedia Office Filing—The MULTOS Approach," C. Thanos, ed., North-Holland, 1990.
[61] Y. Wu, S. Jajodia, and X.S. Wang, "Temporal Database Bibliography Update," available at
[62] M. Vilain, H. Kautz, and P. VanBeek, “Constraint Propagation Algorithms for Temporal Reasoning: A Revised Report,” D.S. Weld and J. de Kleer, eds., Readings in Qualitative Reasoning About Physical Systems, pp. 373–381, Morgan Kaufmann, 1989.
[63] K.H. Weiss, "Formal Specification and Continuos Media," Proc. First Int'l Workshop Network and Operating System Support for Digital Audio and Video, pp. 123-127, Nov. 1990.
[64] D. Woelk, W. Kim, and W. Luther, "An Object-Oriented Approach to Multimedia Databases," Proc. ACM SIGMOD 1986 Int'l Conf. on Management of Data, Vol. 15, No. 2, 1986, ACM Press, New York, pp. 311-325.
[65] W.M. Zuberek, "Performance Evaluation Using Extended Timed Petri-Nets," Proc. Int'l Conf. Timed Petri-Nets, pp. 272-278,Torino, Italy, 1985.

Index Terms:
Multimedia systems, temporal scenario, multimedia synchronization, temporal scenario models, temporal constraints.
Elisa Bertino, Elena Ferrari, "Temporal Synchronization Models for Multimedia Data," IEEE Transactions on Knowledge and Data Engineering, vol. 10, no. 4, pp. 612-631, July-Aug. 1998, doi:10.1109/69.706060
Usage of this product signifies your acceptance of the Terms of Use.