The Community for Technology Leaders
RSS Icon
Subscribe
Issue No.03 - July-Sept. (2012 vol.19)
pp: 24-33
Theodora Tsikrika , University of Applied Sciences Western Switzerland
Jana Kludas , University of Geneva, Switzerland
Adrian Popescu , CEA, LIST, France
ABSTRACT
The ImageCLEF Wikipedia image retrieval task aimed to support ad-hoc image retrieval evaluation using large-scale collections of Wikipedia images and their user-generated annotations.
INDEX TERMS
User-generated content, Internet, Electronic publishing, Image retrieval, Error analysis, Multimedia communication, Wikipedia, Performance evaluation, Large scale systems, Collaboration, incomplete judgments, multimedia, image retrieval evaluation, test collections, topic development, ground truth creation, benchmarks, ImageCLEF, ranking stability, measurement error
CITATION
Theodora Tsikrika, Jana Kludas, Adrian Popescu, "Building Reliable and Reusable Test Collections for Image Retrieval: The Wikipedia Task at ImageCLEF", IEEE MultiMedia, vol.19, no. 3, pp. 24-33, July-Sept. 2012, doi:10.1109/MMUL.2012.17
REFERENCES
1. M. Sanderson, "Test Collection Based Evaluation of Information Retrieval Systems," Foundations and Trends in Information Retrieval, vol. 4, no. 4, 2010, pp. 247–375.
2. H. Müller et al., eds., ImageCLEF: Experimental Evaluation in Visual Information Retrieval, Springer, 2010.
3. T. Tsikrika and J. Kludas, "Overview of the Wikipedia MM task at ImageCLEF 2008," Evaluating Systems for Multilingual and Multimodal Information Access: Proc. 9th Workshop of the Cross-Language Evaluation Forum (CLEF 2008), Revised Selected Papers, LNCS, Springer, 2009, pp. 539–550.
4. T. Tsikrika and J. Kludas, "Overview of the Wikipedia MM task at ImageCLEF 2009," Multilingual Information Access Evaluation Vol. II Multimedia Experiments: Proc. 10th Workshop of the Cross- Language Evaluation Forum (CLEF 2009), Revised Selected Papers, LNCS, Springer, 2010, pp. 60–71.
5. A. Popescu, T. Tsikrika, and J. Kludas, "Overview of the Wikipedia Retrieval Task at ImageCLEF 2010," Working Notes for the CLEF 2010 Workshop, 2010; http://clef2010.org/resources/ proceedings clef2010labs_submission_124.pdf.
6. T. Tsikrika, A. Popescu, and J. Kludas, "Overview of the Wikipedia Image Retrieval Task at ImageCLEF 2011," Working Notes for the CLEF 2011 Labs and Workshop, 2011; http://clef2011.org/resources/proceedings Overview_ImageCLEF_ Wikipeida_Retrieval_Clef2011.pdf.
7. E.M. Voorhees and C. Buckley, "The Effect of Topic Set Size on Retrieval Experiment Error," Proc. 25th ACM SIGIR Conf. Research and Development in Information Retrieval, ACM Press, 2002, pp. 316–323.
8. C. Buckley and E.M. Voorhees, "Retrieval Evaluation with Incomplete Information," Proc. 27th ACM SIGIR Conf. Research and Development in Information Retrieval, ACM Press, 2004, pp. 25–32.
9. J. Zobel, "How Reliable Are the Results of Large-Scale Information Retrieval Experiments?" Proc. 21st ACM SIGIR Conf. Research and Development in Information Retrieval, ACM Press, 1998, pp. 307–314.
10. S. Büttcher et al., "Reliable Information Retrieval Evaluation with Incomplete and Biased Judgments," Proc. 30th ACM SIGIR Conf. Research and Development in Information Retrieval, ACM Press, 2007, pp. 63–70.
11. S. Pal, M. Mitra, and J. Kamps, "Evaluation Effort, Reliability and Reusability in XML Retrieval," J. Am. Soc. for Information Science and Technology, vol. 62, no. 2, 2011, pp. 375–394.
12. T. Westerveld and R. van Zwol, "The INEX 2006 Multimedia Track," Advances in XML Information Retrieval: Proc. 5th Int'l Workshop of the Initiative for the Evaluation of XML Retrieval (INEX 2006), Revised Selected Papers, LNCS, Springer, 2007, pp. 331–344.
13. C. Buckley and E.M. Voorhees, "Evaluating Evaluation Measure Stability," Proc. 23rd ACM SIGIR Conf. Research and Development in Information Retrieval, ACM Press, 2000, pp. 33–40.
14. J.A. Aslam, E. Yilmaz, and V. Pavlu, "The Maximum Entropy Method for Analyzing Retrieval Measures," Proc. 28th ACM SIGIR Conf. Research and Development in Information Retrieval, ACM Press, 2005, pp. 27–34.
15. E. Yilmaz and S. Robertson, "On the Choice of Effectiveness Measures for Learning to Rank," Information Retrieval, vol. 13, no. 3, 2010, pp. 271–290.
16. E. Yilmaz, J.A. Aslam, and S. Robertson, "A New Rank Correlation Coefficient for Information Retrieval," Proc. 31st ACM SIGIR Conf. Research and Development in Information Retrieval, ACM Press, 2008, pp. 587–594.
5 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool