A field study of the relationship and communication between Chinese evaluators and users in thinking aloud usability tests

ABSTRACT

Thinking aloud is the most widely applied usability evaluation method. In order to get reliable usability problems, it is necessary for the evaluators to establish a supportive relationship and communicate effectively with the users. This study investigated the relation and communication between the evaluators and test users in Chinese usability testing sessions. Field observations and interviews were conducted in five companies in Beijing. This research was based mainly on Nisbett's cultural theory and Boren and Ramey's thinking aloud model. The results of the study showed that Chinese users focused mainly on tasks, while evaluators focused on both users and tasks. Further, Chinese users did not think aloud actively; thus, in order to encourage users to speak out, effective communication skills were required for Chinese evaluators. Retrospective thinking aloud and explanation were also used in the tests. Finally, it discussed that communication was appropriate for the formative evaluations, but not for the summative evaluations.

REFERENCES

• 1: Effie Lai-Chong Law , Ebba Thora Hvannberg, Analysis of combinatorial user effect in international usability tests, Proceedings of the SIGCHI conference on Human factors in computing systems, p.9-16, April 24-29, 2004, Vienna, Austria [doi>10.1145/985692.985694]
• 2: Tamler, H., High-tech versus high-touch: The limits of automation in diagnostic usability testing. http://www.htamler.com/papers/techtouch/, 2001.
• 3: Capra, M. G., Usability Problem Description and the Evaluator Effect in Usability Testing. 2006, Virginia Polytechnic Institute and State University: Blacksburg, Virginia. p. 292.
• 4: Jakob Nielsen, Usability Engineering, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1993
• 5: Qingxin Shi , Torkil Clemmensen, Communication patterns and usability problem finding in cross-cultural thinking aloud usability testing, CHI '08 extended abstracts on Human factors in computing systems, April 05-10, 2008, Florence, Italy [doi>10.1145/1358628.1358766]
• 6: Ericsson, K. A. and H. A. Simon, Protocol Analysis. Verbal reports as data. A Bradford Book. 1993: Cambridge Massachusetts.
• 7: Boren, M. T. and J. Ramey, Thinking aloud: Reconciling theory and practice. IEEE Transactions on Professional Communication, 2000. 43(3): p. 261--278.
• 8: Kim, H. S., We talk, therefore we think? A cultural analysis of the effect of talking on thinking. journal of Personality and Social Psychology, 2002. 83(4): p. 828--842.
• 9: Sanchez-Burks, J., R. E. Nisbett, and O. Ybarra, Cultural Styles, Relational Schemas and Prejudice Against Outgroups. 2000, University of Michigan.
• 10: Clemmensen, T. and T. Plocher, The Cultural Usability (CULTUSAB) Project: Studies of Cultural Models in Psychological Usability Evaluation Methods, in Usability and Internationalization - Second International Conference on Usability and Internationalization, HCI International 2007, Beijing, China, Proceedings, Part I, N. Aykin, Editor. Springer: Heidelberg. p. 274--280.
• 11: Alvin Yeo, Cultural effects in usability assessment, CHI 98 conference summary on Human factors in computing systems, p.74-75, April 18-23, 1998, Los Angeles, California, United States [doi>10.1145/286498.286536]
• 12: Faiola, A. and S. A. Matei, Cultural cognitive style and web design: Beyond a behavioral inquiry into computer-mediated communication. Journal of Computer-Mediated Communication, 2005. 11(1).
• 13: Hertzum, M. and N. E. Jacobsen, The evaluator effect: A chilling fact about usability evaluation methods. International Journal of Human-Computer Interaction, 2001. 13(4): p. 421--443.
• 14: Nisbett, R. E., The Geography of Thought. 2003, London: Nicholas Brealey Publishing.
• 15: Nisbett, R. E. and T. Masuda, Cultural and point of view. PNAS 2003. 100(19): p. 11163--11170.
• 16: Ramey, J., et al. Does Think Aloud Work? How Do We Know? in CHI 2006, April 22--27.
• 17: Alvin W. Yeo, Global-software development lifecycle: an exploratory study, Proceedings of the SIGCHI conference on Human factors in computing systems, p.104-111, March 2001, Seattle, Washington, United States [doi>10.1145/365024.365060]
• 18: Hong, Y.-y. and L. M. Mallorie, A dynamic constructivist approach to culture: Lessons learned from personality psychology. Journal of Research in Personality, 2004. 38: p. 59--67.
• 19: Shi, Q. and T. Clemmensen. Relationship Model in Cultural Usability Testing. in HCI International, 2007, Beijing.
• 20: Festinger, L., & Katz, D., Research methods in the behavioral sciences. 1966, New York: Holt, Rinehart and Winston.
• 21: Ross, N., Culture and cognition: implication for theory and method. 2004, Vanderbilt University: International Educational and Professional Publisher.
• 22: Jordan, B. and A. Henderson, Interaction Analysis: Foundations and Practice. The Journal of the Learning Sciences, 1995. 4(1): p. 39--103.
• 23: Punch, K. F., Introduction to social research-quantitative and qualitative approaches. 2005, London: SAGE Publications.
• 24: Torkil Clemmensen , Shi Qingxin, What is part of a usability test?, CHI '08 extended abstracts on Human factors in computing systems, April 05-10, 2008, Florence, Italy [doi>10.1145/1358628.1358898]
• 25: Clemmensen, T., et al. Cultural Usability Tests - How Usability Tests Are Not the Same All over the World. in HCI International 2007. Beijing, China, P: 281--290 (Lecture Notes on Computer Science; 4559).
• 26: Mie Nørgaard , Kasper Hornbæk, What do usability evaluators do in practice?: an explorative study of think-aloud testing, Proceedings of the 6th conference on Designing Interactive systems, June 26-28, 2006, University Park, PA, USA [doi>10.1145/1142405.1142439]
• 27: Zhiwei Guan , Shirley Lee , Elisabeth Cuddihy , Judith Ramey, The validity of the stimulated retrospective think-aloud method as measured by eye tracking, Proceedings of the SIGCHI conference on Human Factors in computing systems, April 22-27, 2006, Montréal, Québec, Canada [doi>10.1145/1124772.1124961]
• 28: Haak, M. J. v.d., M. D. T.d. Jong, and P. J. Schellens, Employing think-aloud protocols and constructive interaction to test the usability of online library catalogues: a methodological comparison. Interacting with Computers 2004. 16: p. 1153--1170.
• 29: Haak, M. J. v.d., M. D. T.d. Jong, and P. J. Schellens, Retrospective vs. concurrent think-aloud protocols: testing the usability of an online library catalogue. BEHAVIOUR & INFORMATION TECHNOLOGY, 2003. 22(5): p. 339--351.
• 30: Carol M. Barnum , Sam Dragga, Usability Testing and Research, Allyn & Bacon, Inc., Needham Heights, MA, 2001
• 31: Laura Leventhal , Julie Barnes, Usability Engineering: Process, Products and Examples, Prentice-Hall, Inc., Upper Saddle River, NJ, 2007
• 32: Yammiyavar, P., T. Clemmensen, and J. Kumar. Analyzing non-verbal cues in Usabiliy Evaluation Tests. in HCI International, 2007, Beijing.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Evaluation/methodology

General Terms:
Human Factors

Keywords:
culture, field study, formative evaluation, thinking aloud usability testing