Crowdsourcing user studies with Mechanical Turk

ABSTRACT

User studies are important for many aspects of the design process and involve techniques ranging from informal surveys to rigorous laboratory studies. However, the costs involved in engaging users often requires practitioners to trade off between sample size, time requirements, and monetary costs. Micro-task markets, such as Amazon's Mechanical Turk, offer a potential paradigm for engaging a large number of users for low time and monetary costs. Here we investigate the utility of a micro-task market for collecting user measurements, and discuss design considerations for developing remote micro user evaluation tasks. Although micro-task markets have great potential for rapidly collecting user measurements at low costs, we found that special care is needed in formulating tasks in order to harness the capabilities of the approach.

REFERENCES

• 1: Benkler, Y. 2002. Coase's penguin,or Linux and the nature of the firm. Yale Law Journal 112, 367--445.
• 2: Morten Sieker Andreasen , Henrik Villemann Nielsen , Simon Ormholt Schrøder , Jan Stage, What happened to remote usability testing?: an empirical study of three methods, Proceedings of the SIGCHI conference on Human factors in computing systems, April 28-May 03, 2007, San Jose, California, USA [doi>10.1145/1240624.1240838]
• 3: BJ Fogg , Jonathan Marshall , Tami Kameda , Joshua Solomon , Akshay Rangnekar , John Boyd , Bonny Brown, Web credibility research: a method for online experiments and early study results, CHI '01 extended abstracts on Human factors in computing systems, March 31-April 05, 2001, Seattle, Washington [doi>10.1145/634067.634242]
• 4: Aniket Kittur , Bongwon Suh , Bryan A. Pendleton , Ed H. Chi, He says, she says: conflict and coordination in Wikipedia, Proceedings of the SIGCHI conference on Human factors in computing systems, April 28-May 03, 2007, San Jose, California, USA [doi>10.1145/1240624.1240698]
• 5: Jared Spool , Will Schroeder, Testing web sites: five users is nowhere near enough, CHI '01 extended abstracts on Human factors in computing systems, March 31-April 05, 2001, Seattle, Washington [doi>10.1145/634067.634236]
• 6: Viégas, F.B., Wattenberg, M., and McKeon, M. 2007. The Hidden Order of Wikipedia. In Proc. of HCI Interactional Conference.
• 7: Wikipedia. Featured Article Criteria. http://en.wikipedia.org/wiki/Wikipedia:Featured_article_criteria, accessed Sep, 2007.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Evaluation/methodology

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.3 Group and Organization Interfaces
          Subjects: Web-based interaction

General Terms:
Experimentation, Human Factors, Measurement

Keywords:
Mechanical Turk, Wikipedia, micro task, remote user study