Toward a best practice for laboratory-based usability evaluations of mobile ICT for hospitals

ABSTRACT

The dynamic nature of clinical work makes it challenging to assess the usability of mobile information and communication technology (ICT) for hospitals. To achieve some of the realism of field evaluations, combined with the control of laboratory-based evaluations, we have conducted usability tests of prototypes in a laboratory custom designed as a full-scale ward section. Nurses and physicians acting out simulated clinical scenarios have used the prototypes.

This paper reports on the general methodological lessons learned from three such formative usability evaluations. We have learned that the physical test environment, the test scenarios, and the prototypes form three variables that need to reflect a sufficient amount of realism and concreteness in order to help generate valid test results. At the same time, these variables are tools that can help control and focus the evaluation on specific issues that one wants to gather data on. We have also learned that encouraging reflection among participants and using detailed multi-perspective recordings of usage can help form a more precise understanding of how mobile ICT can accommodate clinical work.

The current paper aims to inform work toward a best practice for laboratory-based evaluations of mobile ICT for hospitals.

REFERENCES

• 1: Dahl, Y. Ubiquitous Computing at Point of Care in Hospitals: A User-Centered Approach. Doctoral thesis. Norwegian University of Science and Technology (2007).
• 2: Jakob E. Bardram , Claus Bossen, Mobility Work: The Spatial Dimension of Collaboration at a Hospital, Computer Supported Cooperative Work, v.14 n.2, p.131-160, April 2005 [doi>10.1007/s10606-005-0989-y]
• 3: Melanie Kellar , Derek Reilly , Kirstie Hawkey , Malcolm Rodgers , Bonnie MacKay , David Dearman , Vicki Ha , W. Joseph MacInnes , Michael Nunes , Karen Parker , Tara Whalen , Kori M. Inkpen, It's a jungle out there: practical considerations for evaluation in the city, CHI '05 extended abstracts on Human factors in computing systems, April 02-07, 2005, Portland, OR, USA [doi>10.1145/1056808.1056959]
• 4: Tamminen, S., Oulasvirta, A., Toiskallio, K. and Kankainen, A. Understanding Mobile Contexts. Mobile HCI (2003).
• 5: Kjeldskov, J. and Skov, M. B. Studying Usability in Sitro: Simulating Real World Phenomena in Controlled Environments. IJHCI, 27, 1 (2007), 7--37.
• 6: Kjeldskov, J., Skov, M. B., Als, B. S. and Høegh, R. T. Is it Worth the Hassle? Exploring the Added Value of Evaluating the Usability of Context-Aware Mobile Systems in the Field. Mobile HCI (2004), 272--280.
• 7: Christian Monrad Nielsen , Michael Overgaard , Michael Bach Pedersen , Jan Stage , Sigge Stenild, It's worth the hassle!: the added value of evaluating the usability of mobile systems in the field, Proceedings of the 4th Nordic conference on Human-computer interaction: changing roles, p.272-280, October 14-18, 2006, Oslo, Norway [doi>10.1145/1182475.1182504]
• 8: ISO 9241-11 Ergonomic Requirements for Office Work with Visual Display Terminals (VDTs) -- Part 11: Guidance on Usability (1998).
• 9: Joseph E. McGrath, Methodology matters: doing research in the behavioral and social sciences, Human-computer interaction: toward the year 2000, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1995
• 10: Jesper Kjeldskov , Jan Stage, Designing the handheld maritime communicator, Proceedings of the 2003 conference on Designing for user experiences, June 06-07, 2003, San Francisco, California [doi>10.1145/997078.997080]
• 11: Jakob E. Bardram , Thomas R. Hansen, The AWARE architecture: supporting context-mediated social awareness in mobile cooperation, Proceedings of the 2004 ACM conference on Computer supported cooperative work, November 06-10, 2004, Chicago, Illinois, USA [doi>10.1145/1031607.1031639]
• 12: Antti Pirhonen , Stephen Brewster , Christopher Holguin, Gestural and audio metaphors as a means of control for mobile devices, Proceedings of the SIGCHI conference on Human factors in computing systems: Changing our world, changing ourselves, April 20-25, 2002, Minneapolis, Minnesota, USA [doi>10.1145/503376.503428]
• 13: Yngve Dahl , Dag Svanæs, A comparison of location and token-based interaction techniques for point-of-care access to medical information, Personal and Ubiquitous Computing, v.12 n.6, p.459-478, August 2008 [doi>10.1007/s00779-007-0141-8]
• 14: Ole Andre Alsos , Dag Svanæs, Interaction techniques for using handhelds and PCs together in a clinical setting, Proceedings of the 4th Nordic conference on Human-computer interaction: changing roles, p.125-134, October 14-18, 2006, Oslo, Norway [doi>10.1145/1182475.1182489]
• 15: Alsos, O. A. Attention and usability issues in mobile health information systems at point-of-care. MIE (2008).
• 16: Jakob E. Bardram , Claus Bossen, Mobility Work: The Spatial Dimension of Collaboration at a Hospital, Computer Supported Cooperative Work, v.14 n.2, p.131-160, April 2005 [doi>10.1007/s10606-005-0989-y]
• 17: Bødker, S. Scenarios in user-centred design---setting the stage for reflection and action. Interacting with Computers, 13, 1 (2000), 61--75.
• 18: Dag Svanaes , Gry Seland, Putting the users center stage: role playing and low-fi prototyping enable end users to design mobile systems, Proceedings of the SIGCHI conference on Human factors in computing systems, p.479-486, April 24-29, 2004, Vienna, Austria [doi>10.1145/985692.985753]
• 19: Anderson, M. B., Stillman, P. L. and Wang, Y. Growing use of standardized patients in teaching and evaluation in medical education. Teach Learn Med, 6, 1 (1994), 15--22.
• 20: Toward a Framework for Evaluating Ubiquitous Computing Applications, IEEE Pervasive Computing, v.3 n.2, p.82-88, April 2004 [doi>10.1109/MPRV.2004.1316826]
• 21: William L. Bewley , Teresa L. Roberts , David Schroit , William L. Verplank, Human factors testing in the design of Xerox's 8010 “Star” office workstation, Proceedings of the SIGCHI conference on Human Factors in Computing Systems, p.72-77, December 12-15, 1983, Boston, Massachusetts, United States [doi>10.1145/800045.801584]
• 22: Judith Ramey , Ted Boren , Elisabeth Cuddihy , Joe Dumas , Zhiwei Guan , Maaike J. van den Haak , Menno D. T. De Jong, Does think aloud work?: how do we know?, CHI '06 extended abstracts on Human factors in computing systems, April 22-27, 2006, Montréal, Québec, Canada [doi>10.1145/1125451.1125464]
• 23: Kjeldskov, J. and Skov, M. B. Creating Realistic Laboratory Settings: Comparative Studies of Three Think-Aloud Usability Evaluations of a Mobile System. Interact (2003), 663--670.
• 24: Grudin, J. and Pruitt, J. Personas, Participatory Design and Product Development: An Infrastructure for Engagement. Proc. PDC (2002), 144--161.
• 25: Coiera, E.: Four rules for the reinvention of healthcare. British Medical Journal 328 (2004), 1197--1199
• 26: Sørby, I. D., Melby, L. and Nytrø, O. Characterising cooperation in the ward: framework for producing requirements to mobile electronic healthcare records. IJHTM, 7, 6 (2006), 506--521.
• 27: Jakob E. Bardram , Claus Bossen, A web of coordinative artifacts: collaborative work at a hospital ward, Proceedings of the 2005 international ACM SIGGROUP conference on Supporting group work, November 06-09, 2005, Sanibel Island, Florida, USA [doi>10.1145/1099203.1099235]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Evaluation/methodology

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: User-centered design

General Terms:
Design, Experimentation, Human Factors

Keywords:
mobile computing, pervasive computing, point-of-care systems, usability evaluation