This Article 
 Bibliographic References 
 Add to: 
Image Retrieval Using Multiple Evidence Ranking
April 2004 (vol. 16 no. 4)
pp. 408-417

Abstract—The World Wide Web is the largest publicly available image repository and a natural source of attention. An immediate consequence is that searching for images on the Web has become a current and important task. To search for images of interest, the most direct approach is keyword-based searching. However, since images on the Web are poorly labeled, direct application of standard keyword-based image searching techniques frequently yields poor results. In this work, we propose a comprehensive solution to this problem. In our approach, multiple sources of evidence related to the images are considered. To allow combining these distinct sources of evidence, we introduce an image retrieval model based on Bayesian belief networks. To evaluate our approach, we perform experiments on a reference collection composed of 54,000 Web images. Our results indicate that retrieval using an image surrounding text passages is as effective as standard retrieval based on HTML tags. This is an interesting result because current image search engines in the Web usually do not take text passages into consideration. Most important, according to our results, the combination of information derived from text passages with information derived from HTML tags leads to improved retrieval, with relative gains in average precision figures of roughly 50 percent, when compared to the results obtained by the use of each source of evidence in isolation.

[1] Y.A. Aslandogan and C.T. Yu, Techniques and Systems for Image and Video Retrieval IEEE Trans. Knowledge and Data Eng., vol. 11, pp. 56-63, Jan./Feb. 1999.
[2] R. Baeza-Yates and B. Ribeiro-Neto, Modern Information Retrieval, first ed. Addison-Wesley, 1999.
[3] D. Bathurst, R. Bathurst, and D. Davies, The Telling Image: The Changing Balance Between Pictures and Words in a Technological Age, first ed. Clarendon Press, 1990.
[4] A.B. Benitez, M. Beigi, and S.-F. Chang, Using Relevance Feedback in Content-Based Image Metasearch IEEE Internet Computing, vol. 2, pp. 59-69, July 1998.
[5] A.B. Benitez, M. Beigi, and S.F. Chang, MetaSEEk: A Content-Based Meta Search Engine for Images Proc. Storage and Retrieval for Image and Video Databases (SPIE), Dec. 1997.
[6] P. Calado, B. Ribeiro-Neto, N. Ziviani, E. Moura, and I. Silva, Local versus Global Link Information in the Web ACM Trans. Information Systems, vol. 21, no. 1, Jan. 2003.
[7] N.S. Chang and K.S. Fu, Query-by-Pictorial-Example IEEE Trans. Software Eng., vol. 6, pp. 519-524, Nov. 1980.
[8] S.K. Chang and T.L. Kunii, Pictorial Data-Base Systems Computer, vol. 14, pp. 13-21, Nov. 1981.
[9] Z. Chen, L. Wenyin, F. Zhang, M. Li, and H. Zang, Web Mining for Web Image Retrieval J. Am. Soc. Information Science and Technology, vol. 52, pp. 831-839, Aug. 2001.
[10] I.J. Cox, M.L. Miller, T.P. Minka, and T.V. Papathomas, “The Bayesian Image Retrieval System, PicHunter: Theory, Implementation, and Pychophysical Experiments,” IEEE Trans. Image Processing, vol. 9, no. 1, pp. 20-37, 2000.
[11] E.A. El Kwae and M.R. Kabuka, Efficient Content-Based Indexing of Large Image Databases ACM Trans. Information Systems, vol. 18, pp. 171-210, Apr. 2000.
[12] M. Flickner, H. Sawhney, W. Niblack, J. Ashley, Q. Huang, B. Dom, M. Gorkani, J. Hafner, D. Lee, D. Petkovic, D. Steele, and P. Yanker, “Query by Image and Video Content: The QBIC System,” IEEE Computer, 1995.
[13] V. Harmandas, M. Sanderson, and M.D. Dunlop, Image Retrieval by Hypertext Links Proc. 20th Ann. Int'l ACM SIGIR Conf. Research and Development in Information Retrieval, July 1997.
[14] D. Hawking, N. Craswell, and P.B. Thistlewaite, Overview of TREC-7 Very Large Collection Track Proc. Seventh Text REtrieval Conf. (TREC-7), pp. 91-104, Nov. 1998.
[15] D. Hawking, N. Craswell, P.B. Thistlewaite, and D. Harman, Results and Challenges in Web Search Evaluation Computer Networks, vol. 31, pp. 1321-1330, May 1999. Also Proc. Eighth Int'l World Wide Web Conf.
[16] C. Hu, X. Zhu, H. Zhang, and Q. Yang, A Unified Framework for Semantics and Feature Based Relevance Feedback in Image Retrieval Systems Proc. Eighth ACM Int'l Conf. Multimedia, pp. 31-37, Oct. 2000.
[17] C. Jorgensen, Image Access: Bridging Multiple Needs and Multiple Perspectives Introduction and Overview J. Am. Soc. Information Science and Technology, vol. 52, pp. 906-910, Sept. 2001.
[18] M. Kaszkiel, J. Zobel, and R. Sacks-Davis, Efficient Passage Ranking for Document Databases ACM Trans. Information Systems, vol. 17, pp. 406-439, Oct. 1999.
[19] G. Lu and B. William, An Integrated WWW Image Retrieval System Proc. Fifth Australian World Wide Web Conf., Apr. 1999.
[20] V.E. Ogle, “CHABOT—Retrieval from a Relational Database of Images,” Computer, vol. 28, no. 9, pp. 40-48, Sept. 1995.
[21] J. Pearl, Probabilistic Reasoning in Intelligent Systems: Networks of plausible inference, second ed. Morgan Kaufmann, 1988.
[22] B. Ribeiro-Neto and R. Muntz, A Belief Network Model for IR Proc. 19th Ann. Int'l ACM SIGIR Conf. Research and Development in Information Retrieval, pp. 253-260, Aug. 1996.
[23] B. Ribeiro-Neto, I. Silva, and R. Muntz, Bayesian Network Models for IR Soft Computing in Information Retrieval: Techniques and Applications, chapter 11, first ed., pp. 259-291, Springer Verlag, 2000.
[24] Y. Rui, T.S. Huang, and S.-F. Chang, Image Retrieval: Current Techniques, Promising Directions, and Open Issues J. Visual Comm. and Image Representation, vol. 10, pp. 39-62, Mar. 1999.
[25] G. Salton and M.J. McGill, Introduction to Modern Information Retrieval, first ed. McGraw-Hill, 1983.
[26] S. Santini and R. Jain, Integrated Browsing and Querying of Image Databases IEEE Multimedia, vol. 7, no. 3, pp. 26-39, July-Sept. 2000.
[27] S. Sclaroff, L. Taycher, and M. La Cascia, “Imagerover: A Content-Base Image Browser for the World Wide Web,” Proc. Workshop Content-Based Access to Image and Video Libraries, pp. 1,000-1,006, 1997.
[28] I. Silva, B. Ribeiro-Neto, P. Calado, E. Moura, and N. Ziviani, Link-Based and Content-Based Evidential Information in a Belief Network Model Proc. 23rd Ann. Int'l ACM SIGIR Conf. Research and Development in Information Retrieval, pp. 96-103, July 2000.
[29] J.R. Smith and S.-F. Chang, An Image and Video Search Engine for the World-Wide Web Proc. Symp. Electronic Imaging: Science and Technology Storage and Retrieval for Image and Video Databases V, Feb. 1997.
[30] H. Turtle and W.B. Croft, Evaluation of an Inference Network-Based Retrieval Model ACM Trans. Information Systems, vol. 9, pp. 187-222, July 1991.
[31] I.H. Witten, A. Moffat, and T.C. Bell, Managing Gigabytes: Compressing and Indexing Documents and Images, second ed. Morgan Kaufmann, 1999.
[32] S.K.M. Wong and Y.Y. Yao, On Modeling Information Retrieval with Probabilistic Inference ACM Trans. Information Systems, vol. 13, pp. 38-68, Jan. 1995.
[33] J.-K. Wu, Content-Based Indexing of Multimedia Databases IEEE Trans. Knowledge and Data Eng., vol. 9, pp. 978-989, Nov. 1997.
[34] Q. Wu, S.S. Iyengar, and M. Zhu, Web Image Retrieval Using Self-Organizing Feature Map J. Am. Soc. Information Science and Technology, vol. 52, pp. 868-875, Aug. 2001.

Index Terms:
Image retrieval, text-based, Bayesian networks, evidence combination, World Wide Web.
Tatiana Almeida Souza Coelho, P?vel Pereira Calado, Lamarque Vieira Souza, Berthier Ribeiro-Neto, Richard Muntz, "Image Retrieval Using Multiple Evidence Ranking," IEEE Transactions on Knowledge and Data Engineering, vol. 16, no. 4, pp. 408-417, April 2004, doi:10.1109/TKDE.2004.1269666
Usage of this product signifies your acceptance of the Terms of Use.