This Article 
 Bibliographic References 
 Add to: 
A Unified Data Model for Representing Multimedia, Timeline, and Simulation Data
September/October 1998 (vol. 10 no. 5)
pp. 746-767

Abstract—This paper describes a unified data model that represents multimedia, timeline, and simulation data utilizing a single set of related data modeling constructs. A uniform model for multimedia types structures image, sound, video, and long text data in a consistent way, giving multimedia schemas and queries a degree of data independence even for these complex data types. Information that possesses an intrinsic temporal element can all be represented using a construct called a stream. Streams can be aggregated into parallel multistreams, thus providing a structure for viewing multiple sets of time-based information. The unified stream construct permits real-time measurements, numerical simulation data, and visualizations of that data to be aggregated and manipulated using the same set of operators. Prototypes based on the model have been implemented for two medical application domains: thoracic oncology and thermal ablation therapy of brain tumors. Sample schemas, queries, and screenshots from these domains are provided. Finally, a set of examples is included for an accompanying visual query language discussed in detail in another document.

[1] Y. Anzai, A. DeSalles, K. Black, K. Farahani, S. Sinha, D. Castro, and R.B. Lufkin, "Interventional MRI," Radiographics, 1993.
[2] D.R. Aberle, J.D.N. Dionisio, M.F. McNitt-Gray, R.K. Taira, A.F. Cárdenas, J.G. Goldin, K. Brown, R.A. Figlin, and W.W. Chu, "Integrated Multimedia Timeline of Medical Images and Data for Thoracic Oncology Patients," Radiographics, vol. 16, no. 3, pp. 669-681, May 1996.
[3] Y. Anzai, R.B. Lufkin, A. DeSalles, D.R. Hamilton, K. Farahani, and K.L. Black, "Preliminary Experience with MR-Guided Thermal Ablation of Brain Tumors," Am. J. Neuroradiology, vol. 16, no. 1, pp. 39-48 (discussion, pp. 49-52), Jan. 1995.
[4] Apple Computer, Inside Macintosh: Text, Addison-Wesley, Reading, Mass., 1993.
[5] E. Bertino, M. Damiani, and P. Randi, "An Approach to Integrate Multimedia Data in a Knowledge Representation System," T. Catarci, M.F. Costabile, and S. Levialdi, eds., Proc. Int'l Workshop Advanced Visual Interfaces, pp. 109-123,Rome, World Scientific, May 1992.
[6] E. Bertino,M. Negri,G. Pelagatti,, and L. Sbattella,“Object-oriented query languages: The notion and the issues,” IEEE Trans. Knowledge and Data Eng., vol. 4, no. 3, pp. 223-237, 1992.
[7] A.D. Bimbo, E. Vicario, and D. Zingoni, “Symbolic Description and Visual Querying of Image Sequences Using Spatio-Temporal Logic,” IEEE Trans. Knowledge and Data Eng., vol. 7, no. 4, pp. 609-621, Aug. 1995.
[8] W.E. Chu and Q. Chen, “A Structured Approach for Cooperative Query Answering,” IEEE Trans. Knowledge and Data Engineering, Vol. 6 No. 5 Oct. 1994, pp. 738–749.
[9] W.W. Chu and K. Chiang, "Abstraction of High Level Concepts from Numerical Values in Databases," Proc. AAAI Workshop Knowledge Discovery in Databases, 1994.
[10] M. Chock, A.F. Cárdenas, and A. Klinger, "Manipulating Data Structures in Pictorial Information Systems," Computer, pp. 43-50, Nov. 1981.
[11] W.W. Chu, Q. Chen, R.-C. Lee, "Cooperative Query Answering via Type Abstraction Hierarchy," S.M. Deen, ed., Cooperating Knowledge Based Systems, Elsevier, 1990.
[12] Workshop Advances in Data Management for the Scientist and Engineer, W.W. Chu, A.F. Cárdenas and R.K. Taira, eds., National Science Foundation, Boston, Feb. 1993.
[13] W.W. Chu, A.F. Cárdenas, and R.K. Taira, "KMeD: A Knowledge-Based Multimedia Medical Distributed Database System," Information Systems, vol. 20, no. 2, pp. 75-96, 1995.
[14] A.F. Cardenas et al., “The Knowledge-Based Object-Oriented Picquery+Language,” IEEE Trans. Knowledge and Data Eng., Vol. 5, No. 4, 1993, pp. 644–657.
[15] W.W. Chu et al., “A Temporal Evolutionary Object-Oriented Data Model and Its Query Language for Medical Image Management,” Proc. Very Large Data Base Conf., 1992.
[16] Research Foundations in Object-Oriented and Semantic Database Systems, A.F. Cárdenas and Dennis McLeod, eds., Prentice Hall, 1990.
[17] W.W. Chu, M.A. Merzbacher, and L. Berkovich, “The Design and Implementation of CoBase,” SIGMOD Record, vol. 22, no. 2, pp. 517-522, June 1993.
[18] D. Castro, R.E. Saxton, and R.B. Lufkin, "Interstitial Photoablative Laser Therapy Guided by Magnetic Resonance Imaging for the Treatment of Deep Tumors," Seminars of Surgical Oncology, vol. 8, pp. 233-241, 1992.
[19] J.D.N. Dionisio and A.F. Cárdenas, "MQuery: A Visual Query Language for Multimedia, Timeline, and Simulation Data," J. Visual Languages and Computing, vol. 7, pp. 377-401, 1996.
[20] J.D.N. Dionisio and A.F. Cárdenas, A Methodology for User Evaluation of Visual Schema Designers and Query Languages, under review, 1998.
[21] J.D.N. Dionisio, A.F. Cárdenas, R.B. Lufkin, K.L. Black, R.K. Taira, and W.W. Chu, "A Multimedia Database System for Thermal Ablation Therapy of Brain Tumors," J. Digital Imaging, vol. 10, no. 1, pp. 21-26, Feb. 1997.
[22] J.D.N. Dionisio, A.F. Cárdenas, R.K. Taira, D.R. Aberle, W.W. Chu, M.F. McNitt-Gray, J.G. Goldin, and R.B. Lufkin, "A Unified Timeline Model and User Interface for Multimedia Medical Databases," Computerized Medical Imaging and Graphics, vol. 20, no. 4, 1996.
[23] Dept. of Health and Human Services, National Institutes of Health, National Library of Medicine, UMLS Knowledge Sources, Aug. 1992.
[24] D. Le Gall, “MPEG: A Video Compression Standard for Multimedia Applications,” Comm. ACM, Apr. 1991.
[25] S. Gibbs, C. Breiteneder, and D. Tsichritzis, "Data Modeling of Time-Based Media," D. Tsichritzis, ed., Visual Objects, pp. 1-22, Centre Universitaire d'Informatique, Univ. of Geneva, 1993.
[26] A. Gupta, T. Weymouth, and R. Jain, "Semantic Queries With Pictures: The VIMSYS Model," Proc. 17th Int'l Conf. Very Large Data Bases, pp. 69-79, Sept. 1991.
[27] M.R. Harreld, "Brain Aneurysm Blood Flow: Modeling, Simulation, VR Visualization," PhD thesis, Univ. of California, Los Angeles, 1996.
[28] L. Hardman, D.C.A Bulterman, and G. van Rossum, "The Amsterdam Hypermedia Model: Adding Time and Context to the Dexter Model," Comm. ACM, Vol. 37, No. 2, Feb. 1994, pp. 50-62.
[29] I.T. Ieong, "Data Modeling and Query Processing for Image Management," PhD thesis, Univ. of California, Los Angeles, 1993.
[30] H. Ishikawa et al., "The Design and Implementation of an Object-Oriented Multimedia Knowledge Base Management System," ACM Trans. Database Syst., vol. 18, no. 1, pp. 1-50, March 1993.
[31] Proc. NSF Workshop Visual Information Management Systems, R. Jain, ed., Feb. 1992.
[32] P. Keller and M. Keller, Visual Cues: Practical Data Visualization, IEEE Press, Piscataway, N.J., 1993.
[33] V.M. Markowitz and A. Shoshani, "Representing Extended Entity-Relationship Structures in Relational Databases: A Modular Approach," ACM Trans. Database Systems, vol. 17, no. 3, pp. 423-464, Sept. 1992.
[34] E. Oomoto, “Design and Implementation of a Video-Object Database System,” IEEE Trans. Knowledge and Data Eng., vol. 5, no. 4, pp. 629-643, Aug. 1993.
[35] J.M. Pratt and M. Cohen, "A Process-Oriented Scientific Database Model," SIGMOD Record, vol. 21, no. 3, pp. 17-25, Sept. 1992.
[36] E. Rose and A. Segev, "A Temporal Object-Oriented Algebra and Data Model," technical report, Lawrence Berkeley Lab., June 1992.
[37] E. Sciore, "Object Specialization," ACM Trans. Information Systems, vol. 7, no. 2, pp. 103-122, Apr. 1989.
[38] M. Stonebraker et. al., “Tioga: Providing Data Management Support for Scientific Visualization Applications,” Proc. Very Large Data Base Conf., 1993.
[39] D. Swanberg, C.-F. Shu, and R. Jain, "Knowledge Guided Parsing in Video Databases, Proc. Symp. Electronic Imaging: Science and Technology, K.T. Knox and E. Granger, eds., San Jose, Calif., Soc. for Imaging Science and Technology and International Soc. for Optical Engineering, Jan.-Feb. 1993.
[40] Y.T.A. Akutsu, K. Otsuji, and T. Sadakata, “VideoMAP and VideoSpaceIcon: Tools for Anatomizing Video Content,” ACM INTERCHI, 1993, pp. 131-136.
[41] H. Ueda, T. Miyatake, S. Sumino, and A. Nagasaka, "Automatic Structure Visualization for Video Editing," Proc. INTERCHI, S. Ashlund, K. Mullet, A. Henderson, E. Hollnagel, and T. White, eds., pp. 137-141,Amsterdam, ACM, Apr. 1993.

Index Terms:
Multimedia database management, multimedia data model, timeline, simulation, visual data modeling, multimedia data streams, temporal databases, visual querying, multimedia querying, medical multimedia databases.
John David N. Dionisio, Alfonso F. Cárdenas, "A Unified Data Model for Representing Multimedia, Timeline, and Simulation Data," IEEE Transactions on Knowledge and Data Engineering, vol. 10, no. 5, pp. 746-767, Sept.-Oct. 1998, doi:10.1109/69.729730
Usage of this product signifies your acceptance of the Terms of Use.