The Community for Technology Leaders
RSS Icon
Subscribe
Issue No.02 - March-April (2013 vol.17)
pp: 76-81
M. Allahbakhsh , Univ. of New South Wales, Sydney, NSW, Australia
B. Benatallah , Univ. of New South Wales, Sydney, NSW, Australia
A. Ignjatovic , Univ. of New South Wales, Sydney, NSW, Australia
H. R. Motahari-Nezhad , Hewlett-Packard Labs., Palo Alto, CA, USA
ABSTRACT
As a new distributed computing model, crowdsourcing lets people leverage the crowd's intelligence and wisdom toward solving problems. This article proposes a framework for characterizing various dimensions of quality control in crowdsourcing systems, a critical issue. The authors briefly review existing quality-control approaches, identify open issues, and look to future research directions. In the Web extra, the authors discuss both design-time and runtime approaches in more detail.
INDEX TERMS
Crowdsourcing, Communities, Quality control, Encyclopedias, Electronic publishing,crowdsourcing workflows, crowdsourcing, quality control
CITATION
M. Allahbakhsh, B. Benatallah, A. Ignjatovic, H. R. Motahari-Nezhad, E. Bertino, S. Dustdar, "Quality Control in Crowdsourcing Systems: Issues and Directions", IEEE Internet Computing, vol.17, no. 2, pp. 76-81, March-April 2013, doi:10.1109/MIC.2013.20
REFERENCES
1. A. Kittur, E. Chi, and B. Suh, “Crowdsourcing User Studies with Mechanical Turk,” Proc. 26th Ann. SIGCHI Conf. Human Factors in Computing Systems, ACM, 2008, pp. 453–456.
2. R. Khazankin, S. Daniel, and S. Dustdar, “Predicting QoS in Scheduled Crowdsourcing,” Advanced Information Systems Eng., vol. 7328, J. Ralyté et al., eds., Springer, 2012, pp. 460–472.
3. A.J. Quinn and B.B. Bederson, “Human Computation: A Survey and Taxonomy of a Growing Field,” Proc. 2011 Ann. Conf. Human Factors in Computing Systems, ACM, 2011, pp. 1403–1412.
4. D. Schall, F. Skopik, and S. Dustdar, “Expert Discovery and Interactions in Mixed Service-Oriented Systems,” IEEE Trans. Services Computing, vol. 5, no. 2, 2012, pp. 233–245.
5. G. Wang et al., “Serf and Turf: Crowdturfing for Fun and Profit,” Proc. 21st Int'l Conf. World Wide Web, ACM, 2012, pp. 679–688.
6. J.J. Chen, N. Menezes, and A. Bradley, “Opportunities for Crowdsourcing Research on Amazon Mechanical Turk,” Proc. CHI 2011 Workshop Crowdsourcing and Human Computation, 2011; http://crowdresearch.org/chi2011-workshop/ paperschen-jenny.pdf.
7. O. Scekic, H. Truong, and S. Dustdar, “Modeling Rewards and Incentive Mechanisms for Social BPM,” Business Process Management, vol. 7481, A. Barros et al., eds., Springer, 2012, pp. 150–155.
8. E. Agichtein et al., “Finding High-Quality Content in Social Media,” Proc. Int'l Conf. Web Search and Web Data Mining, ACM, 2008, pp. 183–194.
9. P. Crosby, Quality is Free, McGraw-Hill, 1979.
10. A. J⊘sang, R. Ismail, and C. Boyd, “A Survey of Trust and Reputation Systems for Online Service Provision,” Decision Support Systems, vol. 43, no. 2, 2007, pp. 618–644.
11. L.D. Alfaro et al., “Reputation Systems for Open Collaboration,” Comm. ACM, vol. 54, no. 8, 2011, pp. 81–87.
12. S.P. Dow et al., “Shepherding the Crowd Yields Better Work,” Proc. 2012 ACM Conf. Computer Supported Cooperative Work (CSCW 12), ACM, 2012, pp. 1013–1022.
13. A. Kittur et al., “CrowdForge: Crowdsourcing Complex Work,” Proc. 24th Ann. ACM Symp. User Interface Software and Technology, ACM, 2011, pp. 43–52.
14. A. Kulkarni, M. Can, and B. Hartmann, “Collaboratively Crowdsourcing Workflows with Turkomatic,” Proc. 2012 ACM Conf. Computer Supported Cooperative Work (CSCW 12), ACM, 2012, pp. 1003–1012.
15. G. Little et al., “TurKit: Human Computation Algorithms on Mechanical Turk,” Proc. 23nd Ann. ACM Symp. User Interface Software and Technology, ACM, 2010, pp. 57–66.
16. W. Mason and D.J. Watts, “Financial Incentives and the 'Performance of Crowds,'” SIGKDD Explorations Newsletter, vol. 11, 2010, pp. 100–108.
17. H.L. Truong, S. Dustdar, and K. Bhattacharya, “Programming Hybrid Services in the Cloud,” Service-Oriented Computing, vol. 7636, C. Liu et al., eds., Springer, 2012, pp. 96–110.
36 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool