The Community for Technology Leaders
RSS Icon
Subscribe
Issue No.05 - Sept.-Oct. (2012 vol.16)
pp: 28-35
Anand Kulkarni , MobileWorks and University of California, Berkeley
Philipp Gutheim , MobileWorks and University of California, Berkeley
Prayag Narula , MobileWorks and University of California, Berkeley
David Rolnitzky , MobileWorks
Tapan Parikh , University of California, Berkeley
Björn Hartmann , University of California, Berkeley
ABSTRACT
Online labor marketplaces offer the potential to automate tasks too difficult for computers, but don't always provide accurate results. MobileWorks is a crowd platform that departs from the marketplace model to provide robust, high-accuracy results using three new techniques. A dynamic work-routing system identifies expertise in the crowd and ensures that posted work completes within a bounded time with fair wages. A peer-management system helps prevent wrong answers. Last, social interaction techniques let best workers manage and teach other crowd members. This process allows the crowd to collaboratively learn how to solve new tasks.
INDEX TERMS
Accuracy, Humans, Internet, Computer architecture, Routing, Quality control, Redundancy, marketplace, crowdsourcing, human computation, development
CITATION
Anand Kulkarni, Philipp Gutheim, Prayag Narula, David Rolnitzky, Tapan Parikh, Björn Hartmann, "MobileWorks: Designing for Quality in a Managed Crowdsourcing Architecture", IEEE Internet Computing, vol.16, no. 5, pp. 28-35, Sept.-Oct. 2012, doi:10.1109/MIC.2012.72
REFERENCES
1. L. von Ahn and L. Dabbish, "Labeling Images with a Computer Game," Proc. Conf. Human-Computer Interaction (CHI), ACM, 2004, pp. 319–326.
2. G. Little et al., "Exploring Iterative and Parallel Human Computation," Proc. ACM SIGKDD Workshop Human Computation (HCOMP 10), ACM, 2010, pp. 68–76.
3. L. von Ahn, "Games with a Purpose," Computer, vol. 39, no. 6, 2006, pp. 92–94.
4. S. Cooper et al., "Predicting Protein Structures with an Online Game," Nature, vol. 466, 5 Aug. 2010, pp. 756–760.
5. L. von Ahn et al., "reCAPTCHA: Human-Based Character Recognition via Web Security Measures," Science, vol. 321, no. 5895, 2008, pp. 1465–1468.
6. D. Oleson et al., "Programmatic Gold: Targeted and Scalable Quality Assurance in Crowdsourcing," Proc. ACM SIGKDD Workshop on Human Computation (HCOMP 11), ACM, 2011, pp. 43–48.
7. P. Ipeirotis, "Mechanical Turk, Low Wages, and the Market for Lemons," blog, 27 July 2010; www.behind-the-enemy-lines.com/2010/07mechanical-turk-low-wages-and-market.html.
8. M.S. Bernstein et al., "Soylent: A Word Processor with a Crowd Inside," Proc. ACM Symp. User Interface Software and Technology (UIST 10), ACM, 2010, pp. 313–322.
9. A. Kulkarni, M. Can, and B. Hartmann, "Collaboratively Crowdsourcing Workflows with Turkomatic," Proc. Conf. Computer Supported Cooperative Work (CSCW 12), 2012, pp. 1003–1012.
10. S. Dow et al., "Shepherding the Crowd Yields Better Work," Proc. Conf. Computer Supported Cooperative Work (CSCW 12), 2012, pp. 1013–1022.
11. J.P. Bigham et al., "VizWiz: Nearly Real-Time Answers to Visual Questions," Proc. ACM Symp. User Interface Software and Technology (UIST 10), ACM, 2010, pp. 333–342.
12. M. Bernstein et al., "Crowds in Two Seconds: Enabling Realtime Crowd-Powered Interfaces," Proc. ACM Symp. User Interface Software and Technology (UIST 11), ACM, 2011, pp. 33–42.
13. N. Eagle, "txteagle: Mobile Crowdsourcing," Internationalization, Design and Global Development. LNCS 5623, Springer, 2009.
14. S. Khanna et al., "Evaluating and Improving the Usability of Mechanical Turk for Low-Income workers in India," Proc. 1st ACM Symp. Computing for Development (ACM DEV 10), ACM, 2010, pp. 12:1–12:10.
15. B.B. Bederson and A.J. Quinn, "Web Workers Unite! Addressing Challenges of Online Laborers," Proc. Ann. Conf. Extended Abstracts on Human Factors in Computing Systems (CHI EA 11), ACM, 2011, pp. 97–106.
16. M.S. Silberman, L. Irani, and R. Ross, "Ethics and Tactics of Professional Crowdwork," XRDS: Cossroads, vol. 17, no. 2, 2010, pp. 39–43.
17. V. Lehdonvirta and M. Enrkvist, "Knowledge Map of the Virtual Economy," InfoDev, 2011; www.infodev.org/enDocument.1056.pdf.
18. P. Ipeirotis, "Analyzing the Mechanical Turk Marketplace," XRDS: Crossroads, vol. 17, no. 2, 2010, pp. 16–21.
19. J. Ross et al., "Who Are the Crowdworkers? Shifting Demographics in Mechanical Turk," Proc. Int'l Conf. Extended Abstracts on Human Factors in Computing Systems (CHI 10), ACM, 2010, pp. 2863–2872.
38 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool