This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
2012 IEEE 12th International Conference on Data Mining Workshops
Learning from Multiple Annotators: When Data is Hard and Annotators are Unreliable
Brussels, Belgium Belgium
December 10-December 10
ISBN: 978-1-4673-5164-5
The crowd sourcing services became popular making it easy and fast to label datasets by multiple annotators in order to achieve supervised learning tasks. Unfortunately, in this context, annotators are not reliable as they may have different levels of experience or knowledge. Furthermore, the data to be labeled may also vary in their level of difficulty. How do we deal with hard data to label and unreliable annotators? In this paper, we present a probabilistic model to learn from multiple naive annotators, considering that annotators may decline to label an instance when they are unsure. Both errors and ignorance of annotators are integrated separately into the proposed Bayesian model. Experiments on several datasets show that our method achieves superior performance compared to other efficient learning algorithms.
Index Terms:
Supervised learning,Probabilistic logic,Bayesian methods,Approximation algorithms,Reliability,Estimation,Training,Bayesian Analysis,Crowdsourcing,Data Quality,Multiple Annotators,Ignorance
Citation:
Chirine Wolley, Mohamed Quafafou, "Learning from Multiple Annotators: When Data is Hard and Annotators are Unreliable," icdmw, pp.514-521, 2012 IEEE 12th International Conference on Data Mining Workshops, 2012
Usage of this product signifies your acceptance of the Terms of Use.