It's a jungle out there: practical considerations for evaluation in the city

ABSTRACT

An essential aspect of mobile and ubiquitous computing research is evaluation within the expected usage context, including environment. When that environment is an urban center, it can be dynamic, expansive, and unpredictable. Methodologies that focus on genuine use in the environment can uncover valuable insights, although they may also limit measurement and control. In this paper, we present our experiences applying traditional experimental techniques for field research in two separate projects set in urban environments. We argue that although traditional methods may be difficult to apply in cities, the challenges are surmountable, and this kind of field research can be a crucial component of evaluation.

REFERENCES

• 1: Martin Colbert, A diary study of rendezvousing: implications for position-aware computing and communications for the general public, Proceedings of the 2001 International ACM SIGGROUP Conference on Supporting Group Work, September 30-October 03, 2001, Boulder, Colorado, USA [doi>10.1145/500286.500292]
• 2: David Dearman , Kirstie Hawkey , Kori M. Inkpen, Effect of location-awareness on rendezvous behaviour, CHI '05 extended abstracts on Human factors in computing systems, April 02-07, 2005, Portland, OR, USA [doi>10.1145/1056808.1057059]
• 3: Goodman, J., Brewster, S., and Gray, P. (2004). Using Field Experiments to Evaluate Mobile Guides. In Proc. of 3rd Annual Workshop on HCI in Mobile Guides, Glasgow, Scotland.
• 4: Giulio Iacucci , Kari Kuutti, Everyday Life as a Stage in Creating and Performing Scenarios for Wireless Devices, Personal and Ubiquitous Computing, v.6 n.4, p.299-306, September 2002 [doi>10.1007/s007790200031]
• 5: Intille, S. S., Tapia, E. M., Rondoni, J., Beaudin, J., Kukla, C., Agarwal, S., et al. (2003). For Studying Behavior and Technology in Natural Settings. In Proc. of UbiComp 2003, Seattle, WA. 157--174.
• 6: Minna Isomursu , Kari Kuutti , Soili Väinämö, Experience clip: method for user participation and evaluation of mobile concepts, Proceedings of the eighth conference on Participatory design: Artful integration: interweaving media, materials and practices, July 27-31, 2004, Toronto, Ontario, Canada [doi>10.1145/1011870.1011881]
• 7: Kjeldskov, J. and Graham, K. (2003). A Review of Mobile HCI Research Methods. In Proc. of Mobile HCI 2003, Udine, Italy. 317--335.
• 8: Kjeldskov, J., Skov, M. B., Als, B. S., and Hoegh, R. T. (2004). Is It Worth the Hassle? Exploring the Added Value of Evaluating the Usability of Context-Aware Mobile Systems in the Field. In Proc. of Mobile HCI 2004, Glasgow, Scotland.
• 9: Joseph E. McGrath, Methodology matters: doing research in the behavioral and social sciences, Human-computer interaction: toward the year 2000, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1995
• 10: Thomas P. Moran , Leysia Palen , Steve Harrison , Patrick Chiu , Don Kimber , Scott Minneman , William van Melle , Polle Zellweger, “I'll get that off the audio”: a case study of salvaging multimedia meeting records, Proceedings of the SIGCHI conference on Human factors in computing systems, p.202-209, March 22-27, 1997, Atlanta, Georgia, United States [doi>10.1145/258549.258704]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.3 Group and Organization Interfaces
          Subjects: Evaluation/methodology

Keywords:
evaluation, field research, methodology, mobile computing, ubiquitous computing