The Community for Technology Leaders
RSS Icon
Subscribe
Issue No.03 - July-Sept. (2012 vol.19)
pp: 15-23
Martha Larson , Delft University of Technology, The Netherlands
Mohammad Soleymani , Imperial College London, UK
Maria Eskevich , Dublin City University, Ireland
Pavel Serdyukov , Yandex, Moscow
Roeland Ordelman , University of Twente, The Netherlands
Gareth Jones , Dublin City University, Ireland
ABSTRACT
The MediaEval Multimedia Benchmark leveraged community cooperation and crowdsourcing to develop a large Internet video dataset for its Genre Tagging and Rich Speech Retrieval tasks.
INDEX TERMS
Multimedia communication, Benchmark testing, Streaming media, Large-scale systems, Human factors, Speech processing, Algorithm design and analysis, Collaboration, Outsourcing, Internet, crowdsourcing, multimedia, large-scale multimedia data, benchmarking, ground truth
CITATION
Martha Larson, Mohammad Soleymani, Maria Eskevich, Pavel Serdyukov, Roeland Ordelman, Gareth Jones, "The Community and the Crowd: Multimedia Benchmark Dataset Development", IEEE MultiMedia, vol.19, no. 3, pp. 15-23, July-Sept. 2012, doi:10.1109/MMUL.2012.27
REFERENCES
1. P. Over et al., "Creating a Web-Scale Video Collection for Research," Proc. ACM MM'09 Workshop on Web-Scale Multimedia Corpus (WSMC), ACM Press, 2009, pp. 25–32.
2. G. Kossinets, "Effects of Missing Data in Social Networks," Social Networks, vol. 28, no. 3, 2006, pp. 247–268.
3. B.R. Rowe et al., "Economic Impact Assessment of NIST's Text Retrieval Conference (TREC) Program: Final Report," Nat'l Inst. Standards and Technology, 2010; http://trec.nist.gov/pubs2010.economic.impact.pdf.
4. C.V. Thornley et al., "The Scholarly Impact of TRECVid (2003–2009)," J. Am Soc. for Information Science and Technology, vol. 62, no. 4, 2011, pp. 613–627.
5. A.F. Smeaton, P. Over, and W. Kraaij, "Evaluation Campaigns and TRECVid," Proc. ACM Int'l Conf. Multimedia Information Retrieval, ACM Press, 2006, pp. 321–330.
6. C.W. Cleverdon, Report on the Testing and Analysis of an Investigation into the Comparative Efficiency of Indexing Systems, tech. report, Cranfield Coll. of Aeronautics, 1962; http://dspace.lib.cranfield.ac.uk/handle/ 1826836.
7. L. von Ahn and L. Dabbish, "Labeling Images with a Computer Game," Proc. ACM SIGCHI Conf. Human factors in Computing Systems (CHI), ACM Press, 2004, pp. 319–326.
8. S. Nowak and S. Rüger, "How Reliable Are Annotations via Crowdsourcing: A Study about Inter-annotator Agreement for Multi-label Image Annotation," Proc. ACM Int'l Conf. Multimedia Information Retrieval (MIR), 2010, pp. 557–566.
9. O. Alonso, D.E. Rose, and B. Stewart, "Crowdsourcing for relevance evaluation," SIGIR Forum, vol. 42, no. 2, 2008, pp. 9–15.
10. J.-I. Biel, O. Aran, and D. Gatica-Perez, "You Are Known By How You Vlog: Personality Impressions and Nonverbal Behavior in YouTube," Proc. 5th Int'l AAAI Conf. Weblogs and Social Media (ICWSM), AAAI Press, 2011, pp. 446–449.
11. M. Soleymani and M. Larson, "Crowdsourcing for Affective Annotation of Video: Development of a Viewer-reported Boredom Corpus," Proc. SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation, 2010; www.ischool.utexas.edu/~cse2010CSE2010-Proceedings.pdf.
12. N. Ramzan et al., "The Participation Payoff: Challenges and Opportunities for Multimedia Access in Networked Communities," Proc. ACM Int'l Conf. Multimedia Information Retrieval (MIR), ACM Press, 2010, pp. 487–496.
13. M. Larson et al., eds., MediaEval Benchmark 2011: MediaEval 2011 Benchmark Workshop, 2011; http://ceur-ws.orgVol-807.
14. P. Kelm, S. Schmiedeke, and T. Sikora, "Feature-Based Video Key Frame Extraction for Low Quality Video Sequences," Proc. 10th Workshop on Image Analysis for Multimedia Interactive Services (WIAMIS), IEEE CS Press, 2009, pp. 25–28.
15. L. Lamel and J.-L. Gauvain, "Speech Processing for Audio Indexing" Advances in Natural Language Processing, LNCS 5221, Springer, 2008, pp. 4–15.
16. M. Larson et al., "Automatic Tagging and Geotagging in Video Collections and Communities," Proc. 1st ACM Int'l Conf. Multimedia Retrieval (ICMR), ACM Press, 2011, article no. 51.
17. B. Ionescu et al., "Content-Based Video Description for Automatic Video Genre Categorization," Proc. 18th Int'l Conf. Advances in Multimedia Modeling (MMM), K. Schoeffmann et al., eds., LNCS 7131, Springer, 2012, pp. 51-62.
18. A. Stolcke et al., "Dialogue Act Modeling for Automatic Tagging and Recognition of Conversational Speech," J. Computational Linguistics, vol. 26, no. 3, 2000, pp. 339–373.
27 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool