This Article 
 Bibliographic References 
 Add to: 
How to Complete Performance Graphs in Content-Based Image Retrieval: Add Generality and Normalize Scope
February 2005 (vol. 27 no. 2)
pp. 245-251
The performance of a Content-Based Image Retrieval (CBIR) system, presented in the form of Precision-Recall or Precision-Scope graphs, offers an incomplete overview of the system under study: The influence of the irrelevant items (embedding) is obscured. In this paper, we propose a comprehensive and well-normalized description of the ranking performance compared to the performance of an Ideal Retrieval System defined by ground-truth for a large number of predefined queries. We advocate normalization with respect to relevant class size and restriction to specific normalized scope values (the number of retrieved items). We also propose new three and two-dimensional performance graphs for total recall studies in a range of embeddings.

[1] C. Baumgarten, “A Probabilistic Solution to the Selection and Fusion Problem in Distributed Information Retrieval,” Proc. Int'l ACM Special Interest Groupon Information Retrieval Conf. (SIGIR '99), pp. 246-253, 1999.
[2] I.J. Cox, M.L. Miller, T. Minka, T.V. Papathomas, and P.N. Yianilos, “The Bayesian Image Retrieval System, PicHunter: Theory, Implementation, and Psychophysical Experiments,” IEEE Trans. Image Processing, vol. 9, no. 1, pp. 20-37, 2000.
[3] D.P. Huijsmans and N. Sebe, “Extended Performance Graphs for Cluster Retrieval,” Proc. IEEE Int'l Conf. Computer Vision and Pattern Recognition (CVPR '01), pp. 26-31, 2001.
[4] D.P. Huijsmans and N. Sebe, “Content-Based Indexing Performance: A Class Size Normalized Precision, Recall, Generality Evaluation,” Proc. IEEE Int'l Conf. Image Processing (ICIP '03), pp. 733-736, 2003.
[5] The SMART Retrieval System, G. Salton, ed. Prentice Hall, 1971.
[6] D.V. Gokhale and S. Kullback, The Information in Contingency Tables. New York: Marcel Dekker, Inc., 1978.
[7] http:/, 2004.
[8] D.P. Huijsmans, N. Sebe, and M.S. Lew, “A Ground-Truth Training Set for Hierarchical Clustering in Content-Based Image Retrieval,” Proc. Fourth Int'l Conf. Advances in Visual Information Systems, pp. 500-510, 2000.
[9] IEEE Trans. Circuits and Systems for Video Technology, special issue: MPEG-7, vol. 11, no. 6, 2001.
[10] H. Müller, S. Marchand-Maillet, and T. Pun, “The Truth about Corel— Evaluation in Image Retrieval,” Proc. Int'l Conf. Image and Video Retrieval (CIVR '02), pp. 38-49, 2002.
[11] H. Müller, W. Müller, D.M. Squire, S. Marchand-Maillet, and T. Pun, “Performance Evaluation in Content-Based Image Retrieval: Overview and Proposals,” Pattern Recognition Letters, vol. 22, pp. 593-601, 2001.
[12] K. Porkaew, K. Chakrabarti, and S. Mehrotra, “Query Refinement for Multimedia Similarity Retrieval in MARS,” ACM Multimedia, pp. 235-238, 1999.
[13] V. Raghavan, P. Bollmann, and G.S. Jung, “A Critical Investigation of Recall and Precision as Measures of Retrieval System Performance,” ACM Trans. Information Systems, vol. 7, pp. 205-229, 1989.
[14] S.E. Robertson, “The Parametric Description of Retrieval Tests. Part II: Overall Measures,” J. Documentation, vol. 25, no. 2, pp. 93-107, 1969.
[15] J.A. Swets, “Information Retrieval Systems,” Science, vol. 141, pp. 245-250, 1963.
[16] J. Tague-Sutcliffe, “The Pragmatics of Information Retrieval Experimentation, Revisited,” Information Processing and Management, vol. 28, no. 4, pp. 467-490, 1992.
[17] Handbook of Medical Informatics, J.H. van Bemmel and M.A. Musen, eds. Springer, 1997.
[18] C.J. van Rijsbergen, Information Retrieval. London: Butterworths, 1979.
[19] N. Vasconcelos and A. Lippman, “A Probabilistic Architecture for Content-Based Image Retrieval,” Proc. Computer Vision and Pattern Recognition, pp. 216-221, 2000.
[20] Proc. Text REtrieval Conf., E.M. Voorhees and D. Harman, eds., 1999.

Index Terms:
Multimedia information systems, information retrieval, content-based image retrieval, performance evaluation.
Dionysius P. Huijsmans, Nicu Sebe, "How to Complete Performance Graphs in Content-Based Image Retrieval: Add Generality and Normalize Scope," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 27, no. 2, pp. 245-251, Feb. 2005, doi:10.1109/TPAMI.2005.30
Usage of this product signifies your acceptance of the Terms of Use.