This Article 
 Bibliographic References 
 Add to: 
Knowledge-Based Image Retrieval with Spatial and Temporal Constructs
November/December 1998 (vol. 10 no. 6)
pp. 872-888

Abstract—A knowledge-based approach to retrieve medical images by feature and content with spatial and temporal constructs is developed. Selected objects of interest in a medical image (e.g., x-ray, MR image) are segmented, and contours are generated from these objects. Features (e.g., shape, size, texture) and content (e.g., spatial relationships among objects) are extracted and stored in a feature and content database. Knowledge about image features can be expressed as a hierarchical structure called a Type Abstraction Hierarchy (TAH). The high-level nodes in the TAH represent more general concepts than low-level nodes. Thus, traversing along TAH nodes allows approximate matching by feature and content if an exact match is not available. TAHs can be generated automatically by clustering algorithms based on feature values in the databases and hence are scalable to large collections of image features. Further, since TAHs are generated based on user classes and applications, they are context- and user-sensitive. A knowledge-based semantic image model is proposed that consists of four layers (raw data layer, feature and content layer, schema layer, and knowledge layer) to represent the various aspects of an image objects' characteristics. The model provides a mechanism for accessing and processing spatial, evolutionary, and temporal queries. A knowledge-based spatial temporal query language (KSTL) has developed that extends ODMG's OQL and supports approximate matching of feature and content, conceptual terms, and temporal logic predicates. Further, a visual query language has been developed that accepts point-click-and-drag visual iconic input on the screen that is then translated into KSTL. User models are introduced to provide default parameter values for specifying query conditions. We have implemented a Knowledge-Based Medical Database System (KMeD) at UCLA, and it is currently under evaluation by the medical staff. The results from this research should be applicable to other multimedia information systems as well.

[1] M. Arya, W. Cody, C. Faloutsos, J. Richardson, and A. Toga, "QBISM: A Prototype 3D Medical Image Database System," Bull. Technical Committee Data Eng., vol. 16, no. 1, pp. 38-42, Mar. 1993.
[2] J.F. Allen, “Maintaining Knowledge about Temporal Intervals,” Comm. ACM, vol. 26, no. 11, pp. 832–843, 1983.
[3] J.R. Bach, S. Paul, and R. Jain, “A Visual Information Management System for the Interactive Retrieval of Faces,” IEEE Trans. Knowledge and Data Eng., vol. 5, no. 4, pp. 619-628, 1993.
[4] A.D. Bimbo, E. Vicario, and D. Zingoni, “Symbolic Description and Visual Querying of Image Sequences Using Spatio-Temporal Logic,” IEEE Trans. Knowledge and Data Eng., vol. 7, no. 4, pp. 609-621, Aug. 1995.
[5] "The Object Database Standard: ODMG-93 (Release 1.2)," R.G.G. Cattell, ed., Morgan Kaufmann, 1996.
[6] W.E. Chu and Q. Chen, “A Structured Approach for Cooperative Query Answering,” IEEE Trans. Knowledge and Data Engineering, Vol. 6 No. 5 Oct. 1994, pp. 738–749.
[7] W.W. Chu, K. Chiang, C.C. Hsu, and H. Yau, "An Error-Based Conceptual Clustering Method for Providing Approximate Query Answers," Comm. ACM, Virtual Extension Ed., Dec. 1996, url:
[8] W.W. Chu, A.F. Cárdenas, and R.K. Taira, "KMeD: A Knowledge-Based Multimedia Medical Distributed Database System," Information Systems, vol. 20, no. 2, pp. 75-96, 1995.
[9] F. Cuppens and R. Demolombe, "How to Recognize Interesting Topics to Provide Cooperative Answering," Information Systems, Vol. 14, No. 2, 1989, pp. 163-173.
[10] A.F. Cardenas et al., “The Knowledge-Based Object-Oriented Picquery+Language,” IEEE Trans. Knowledge and Data Eng., Vol. 5, No. 4, 1993, pp. 644–657.
[11] W.W. Chu, T. Ieong, and R. Taira, "A Semantic Modeling Approach for Image Retrieval by Content," VLDB J., vol. 3, no. 4,445-478, Oct. 1994.
[12] I.F. Cruz, "Doodle," Proc. SIGMOD, 1992.
[13] S.K. Chang, C.W. Yan, D.C. Dimitroff, and T. Arndt, “An Intelligent Image Database System,” IEEE Trans. Software Eng., vol. 14, no. 5, pp. 681-688, May 1988.
[14] J.D.N. Dionisio and A.F. Cárdenas, "MQuery: A Visual Query Language for Multimedia, Timeline, and Simulation Data," J. Visual Languages and Computing, special issue on image and video databases: visual browsing, querying, and retrieval, vol. 7, no. 4, pp. 377-401, 1996.
[15] J.D. Dionisio and A.F. Cardenas, “A Unified Model for Representing Multimedia, Timeline and Simulation Data,” IEEE Trans. Knowledge and Data Eng., vol. 10, no. 5, Sept./Oct. 1998.
[16] Y.F. Day, S. Dagtas, M. Iino, A. Khokhar, and A. Ghafoor, “Object-Oriented Conceptual Modeling of Video Data,” Proc. Data Eng. (DE '95), pp. 401-408, 1995.
[17] D. Daneels, D. Van Campenhout et al., "Interactive Outlining: An Improved Approach Using Active Contours," , Image and Video Storage and Retrieval, SPIE, 1993.
[18] E. Emerson, "Temporal and Modal Logic," Handbook of Theoretical Computer Science, J. Leeuwen, ed., chapter 16, pp. 995-1, 072. Elsevier, 1990.
[19] N. Roussopoulos et al., “An Efficient Pictorial Database System for PSQL,” IEEE Trans. Software Eng., vol. 14, no. 5, pp. 639-651, May 1988.
[20] D. Gabbay, "The Declarative Past and Imperative Future Temporal Logic in Specification: Altrincham Workshop," Lecture Notes in Computer Science 398, Springer-Verlag, 1989.
[21] S. Gibbs, C. Breiteneder, and D. Tsichritzis, "Data Modeling of Time-Based Media," D. Tsichritzis, ed., Visual Objects, Centre Universitaire d'Informatique, Univ. of Geneva, pp. 1-22, 1993.
[22] D. Gabbay and P. McBrien, "Temporal Logic and Historical Databases," Proc. Conf. Very Large Data Bases, 1991.
[23] W.E.L. Grimson, Object Recognition by Computer. MIT Press, 1990.
[24] C.C. Hsu, W.W. Chu, and R.K. Taira, “A Knowledge-Based Approach for Retrieving Images by Content,” IEEE Trans. Knowledge and Data Eng., vol. 8, no. 4, pp. 522-532, 1996.
[25] H.K. Huang and R.K. Taira, "Infrastructure Design of a Picture Archiving and Communication System," Am. J. Roentgenology, vol. 158, pp. 743-749, 1992.
[26] M.D. Levine, Vision in Man and Machine, McGraw-Hill, 1985.
[27] T.D.C. Little and A. Ghafoor, “Interval-Based Conceptual Models for Time-Dependent Multimedia Data,” IEEE Trans. Knowledge and Data Eng., vol. 5, no. 4, pp. 551-563, Aug. 1993.
[28] B.J. Liu, R.K. Taira, J. Shim, and P. Keaton, "Automatic Segmentation of Bones from Digital Hand Radiographs," Proc. SPIE: Medical Imaging Image Processing, vol. 2,434, pp. 659-669, Feb. 1995.
[29] W.N. Martin and J.K. Aggarwal, "Computer Analysis of Dynamic Scenes Containing Currilinear Figures," Pattern Recognition, vol. 11, pp. 169-178, 1979.
[30] L. Mohan and R.L. Kashyap, "A Visual Query Language for Graphical Interaction with Schema-Intensive Databases," IEEE Trans. Knowledge and Data Eng., vol. 5, no. 5, pp. 843-858, Oct. 1993.
[31] R.S. Michalski and R.E. Stepp, "Learning from Observation: Conceptual Clustering," Machine Learning, R.S. Michalski, J.G. Carbonell, and T.M. Mitchell, eds., vol. 1, Morgan Kaufmann, 1983.
[32] W. Niblack, R. Barber et al., , "The QBIC Project: Querying Images by Content Using Color, Texture, and Shape," Storage and Retrieval for Images and Video Databases, SPIE, 1993.
[33] A. Pnueli, "The Temporal Logic of Programs," Proc. 18th Ann. IEEE Symp. Foundations of Computer Science, 1977.
[34] J. Richardson, "Supporting Lists in a Data Model (A Timely Approach)," Proc. ACM SIGMOD, 1992.
[35] B. Subramanian, T.W. Leung, S.L. Vandenberg, and S.B. Zdonik, "The Aqua Approach to Querying Lists and Trees in Object-Oriented Databases," Proc. IEEE Int'l Conf. Data Eng., 1995.
[36] Temporal Databases, A. Tansel, J. Clifford, S. Gadia, S. Jajodia, A. Segev, and R. Snodgrass, eds., Benjamin/Cummings, 1993.
[37] R. Wasserman, R. Acharya et al., , "Multimodality Tumor Delineation via Fuzzy Fusion and Deformable Modelling," Proc. SPIE Medical Image: Image Processing, 1995.
[38] A.J. Worth, S. Lehar, and D.N. Kennedy, "A Recurrent Cooperative/Competitive Field for Segmentation of Magnetic Resonance Brain Images," IEEE Trans. Knowledge and Data Eng., vol. 4, no. 2, pp. 156-161, Apr. 1992.
[39] W.J. Weiland and B. Shneiderman, "A Graphical Query Interface Based on Aggregation/Generalization Hierarchies," Information Systems, 1993.

Index Terms:
Image database systems, visual query language, multimedia data modeling, knowledge-based query processing, temporal and spatial data modeling medical images, cooperative query answering content based image retrieval.
Wesley W. Chu, Chih-Cheng Hsu, Alfonso F. Cárdenas, Ricky K. Taira, "Knowledge-Based Image Retrieval with Spatial and Temporal Constructs," IEEE Transactions on Knowledge and Data Engineering, vol. 10, no. 6, pp. 872-888, Nov.-Dec. 1998, doi:10.1109/69.738355
Usage of this product signifies your acceptance of the Terms of Use.