Behind the curtain: lessons learned from a Wizard of Oz field experiment

ABSTRACT

Large scale, organization-wide groupware systems are high risk development efforts. Requirements gathering and early evaluation are constrained by the need for the system to attain a critical mass of users and content. One approach to mitigate this risk is to employ Wizard of Oz style system simulations during the requirements gathering phase. While this method has historically been used to test quasi-functional system prototypes, we have found it to be a useful method for assessing organizational feasibility. After introducing this adaptation of the Wizard of Oz method, our approach is grounded with a case study in which it is was successfully used as a proof-of-concept requirements gathering exercise for an expertise recommender and organizational memory system project. Transferable lessons learned are distilled and discussed.

REFERENCES

• 1: Mark S. Ackerman, Augmenting the organizational memory: a field study of answer garden, Proceedings of the 1994 ACM conference on Computer supported cooperative work, p.243-252, October 22-26, 1994, Chapel Hill, North Carolina, United States [doi>10.1145/192844.193019]
• 2: Mark S. Ackerman , David W. McDonald, Answer Garden 2: merging organizational memory with collaborative help, Proceedings of the 1996 ACM conference on Computer supported cooperative work, p.97-105, November 16-20, 1996, Boston, Massachusetts, United States [doi>10.1145/240080.240203]
• 3: Nils Dahlbäck , Arne Jönsson , Lars Ahrenberg, Wizard of Oz studies: why and how, Proceedings of the 1st international conference on Intelligent user interfaces, p.193-200, January 04-07, 1993, Orlando, Florida, United States [doi>10.1145/169891.169968]
• 4: Leonard N. Foner, Yenta: a multi-agent, referral-based matchmaking system, Proceedings of the first international conference on Autonomous agents, p.301-307, February 05-08, 1997, Marina del Rey, California, United States [doi>10.1145/267658.267732]
• 5: John D. Gould , John Conti , Todd Hovanyecz, Composing letters with a simulated listening typewriter, Communications of the ACM, v.26 n.4, p.295-308, April 1983 [doi>10.1145/2163.358100]
• 6: Jonathan Grudin, Groupware and social dynamics: eight challenges for developers, Communications of the ACM, v.37 n.1, p.92-105, Jan. 1994 [doi>10.1145/175222.175230]
• 7: Scott Hudson , James Fogarty , Christopher Atkeson , Daniel Avrahami , Jodi Forlizzi , Sara Kiesler , Johnny Lee , Jie Yang, Predicting human interruptibility with sensors: a Wizard of Oz feasibility study, Proceedings of the SIGCHI conference on Human factors in computing systems, April 05-10, 2003, Ft. Lauderdale, Florida, USA [doi>10.1145/642611.642657]
• 8: Scott R. Klemmer , Anoop K. Sinha , Jack Chen , James A. Landay , Nadeem Aboobaker , Annie Wang, Suede: a Wizard of Oz prototyping tool for speech user interfaces, Proceedings of the 13th annual ACM symposium on User interface software and technology, p.1-10, November 06-08, 2000, San Diego, California, United States [doi>10.1145/354401.354406]
• 9: Wayne G. Lutters , Mark S. Ackerman, Achieving safety: a field study of boundary objects in aircraft technical support, Proceedings of the 2002 ACM conference on Computer supported cooperative work, November 16-20, 2002, New Orleans, Louisiana, USA [doi>10.1145/587078.587116]
• 10: David Maulsby , Saul Greenberg , Richard Mander, Prototyping an intelligent agent through Wizard of Oz, Proceedings of the SIGCHI conference on Human factors in computing systems, p.277-284, April 24-29, 1993, Amsterdam, The Netherlands [doi>10.1145/169059.169215]
• 11: David W. McDonald , Mark S. Ackerman, Expertise recommender: a flexible recommendation system and architecture, Proceedings of the 2000 ACM conference on Computer supported cooperative work, p.231-240, December 2000, Philadelphia, Pennsylvania, United States [doi>10.1145/358916.358994]
• 12: Lennart Molin, Wizard-of-Oz prototyping for co-operative interaction design of graphical user interfaces, Proceedings of the third Nordic conference on Human-computer interaction, p.425-428, October 23-27, 2004, Tampere, Finland [doi>10.1145/1028014.1028086]
• 13: Jenny Preece , Yvonne Rogers , Helen Sharp, Interaction Design, John Wiley & Sons, Inc., New York, NY, 2002
• 14: Paul Resnick , Neophytos Iacovou , Mitesh Suchak , Peter Bergstrom , John Riedl, GroupLens: an open architecture for collaborative filtering of netnews, Proceedings of the 1994 ACM conference on Computer supported cooperative work, p.175-186, October 22-26, 1994, Chapel Hill, North Carolina, United States [doi>10.1145/192844.192905]
• 15: Loren Terveen , Will Hill , Brian Amento , David McDonald , Josh Creter, PHOAKS: a system for sharing recommendations, Communications of the ACM, v.40 n.3, p.59-62, March 1997 [doi>10.1145/245108.245122]
• 16: White, Kevin F. Leveraging Organizationally Adjacent Expertise. MS Thesis, University of Maryland Baltimore County. (2004).

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.3 Group and Organization Interfaces
          Subjects: Computer-supported cooperative work

Keywords:
Wizard of Oz, expertise identification, field experiment, human factors, information reuse, study design