Issue No. 05 - Sept.-Oct. (2012 vol. 16)
DOI Bookmark: http://doi.ieeecomputersociety.org/10.1109/MIC.2012.71
Jeroen B.P. Vuurens , The Hague University of Applied Science
Arjen P. de Vries , Centrum Wiskunde & Informatica
The performance of information retrieval (IR) systems is commonly evaluated using a test set with known relevance. Crowdsourcing is one method for learning the relevant documents to each query in the test set. However, the quality of relevance learned through crowdsourcing can be questionable, because it uses workers of unknown quality with possible spammers among them. To detect spammers, the authors' algorithm compares judgments between workers; they evaluate their approach by comparing the consistency of crowdsourced ground truth to that obtained from expert annotators and conclude that crowdsourcing can match the quality obtained from the latter.
Accuracy, Internet, Conferences, Information retrieval, Unsolicited electronic mail, Detection algorithms, Reliability, spam, crowdsourcing, judgment, quality, relevance
J. B. Vuurens and A. P. de Vries, "Obtaining High-Quality Relevance Judgments Using Crowdsourcing," in IEEE Internet Computing, vol. 16, no. , pp. 20-27, 2012.