Usability testing of notification interfaces: are we focused on the best metrics?

ABSTRACT

Notification interfaces that continuously present peripheral information have received increasing interest within the HCI community, especially those supporting awareness of others' activities. While recent empirical studies have focused on information design aspects of peripheral displays, there have been few reported studies that comparatively evaluate actual systems. To this end, this article describes our efforts in comparing three interfaces that inform a remote user about activities within a given setting. Our data allow conclusions about comparative interface usability and preference, and provide an indication about metrics that are valuable to focus on in evaluations for these types of interfaces. In particular, we find that quantitative, performance related metrics, such as the correctness of notification interpretation and interruption to a primary task, are much less conclusive for fully implemented peripheral interfaces than qualitative judgments based on the usage experience.

REFERENCES

• 1: Bartram, L., Ware, C., & Calvert, T. (2001), Moving icons: Detection and distraction, in Proceedings of the IFIP TC.13 International Conference on Human-Computer Interaction (INTERACT '01), IOS Press, pp.157--165.
• 2: Richard Bowden , Pakorn Kaewtrakulpong , Martin Lewin, Jeremiah: the face of computer vision, Proceedings of the 2nd international symposium on Smart graphics, p.124-128, June 11-13, 2002, Hawthorne, New York [doi>10.1145/569005.569023]
• 3: Michael Boyle , Christopher Edwards , Saul Greenberg, The effects of filtered video on awareness and privacy, Proceedings of the 2000 ACM conference on Computer supported cooperative work, p.1-10, December 2000, Philadelphia, Pennsylvania, United States [doi>10.1145/358916.358935]
• 4: J. J. Cadiz , Gina Venolia , Gavin Jancke , Anoop Gupta, Designing and deploying an information awareness interface, Proceedings of the 2002 ACM conference on Computer supported cooperative work, November 16-20, 2002, New Orleans, Louisiana, USA [doi>10.1145/587078.587122]
• 5: Catrambone, R. & Stasko, J., & Xiao, J. (2002), Anthropomorphic agents as a user interaface paradigm: Experimental findings and a framework for research, in W. D. Gray & C. D. Schumm (eds.), Proceedings of the Twenty-fourth Annual Conference of the Cognitive Science Society (CogSci '02), Erlbaum, pp. 166--171.
• 6: C. M. Chewar , D. Scott McCrickard , Ali Ndiwalana , Chris North , Jon Pryor , David Tessendorf, Secondary task display attributes: optimizing visualizations for cognitive task suitability and interference avoidance, Proceedings of the symposium on Data Visualisation 2002, May 27-29, 2002, Barcelona, Spain
• 7: Cutrell, E., Czerwinski, M., Horvitz, E. (2001), Notification, disruption, and memory: Effects of messaging interruptions on memory and performance, in Proceedings of the IFIP TC.13 International Conference on Human-Computer Interaction (INTERACT '01), IOS Press,, IOS Press, pp. 263--269.
• 8: Paul Dourish , Sara Bly, Portholes: supporting awareness in a distributed work group, Proceedings of the SIGCHI conference on Human factors in computing systems, p.541-547, May 03-07, 1992, Monterey, California, United States [doi>10.1145/142750.142982]
• 9: Thomas Erickson , Wendy A. Kellogg, Social translucence: an approach to designing systems that support social processes, ACM Transactions on Computer-Human Interaction (TOCHI), v.7 n.1, p.59-83, March 2000 [doi>10.1145/344949.345004]
• 10: Greenberg, S. & Kuzuoka, H. (2000), Using digital but physical surrogates to mediate awareness, communication and privacy in media spaces, Personal Technologies 4(1), 1--17.
• 11: Scott E. Hudson , Ian Smith, Techniques for addressing fundamental privacy and disruption tradeoffs in awareness support systems, Proceedings of the 1996 ACM conference on Computer supported cooperative work, p.248-257, November 16-20, 1996, Boston, Massachusetts, United States [doi>10.1145/240080.240295]
• 12: McCrickard, D. S., Catrambone, R., Stasko, J. T. (2001), Evaluating animation in the periphery as a mechanism for maintaining awareness in Proceedings of the IFIP TC.13 International Conference on Human-Computer Interaction (INTERACT '01), IOS Press, pp. 558--565.
• 13: D. Scott McCrickard , C. M. Chewar, Attuning notification design to user goals and attention costs, Communications of the ACM, v.46 n.3, March 2003 [doi>10.1145/636772.636800]
• 14: Elin Rønby Pedersen , Tomas Sokoler, AROMA: abstract representation of presence supporting mutual awareness, Proceedings of the SIGCHI conference on Human factors in computing systems, p.51-58, March 22-27, 1997, Atlanta, Georgia, United States [doi>10.1145/258549.258584]
• 15: Johan Redström , Tobias Skog , Lars Hallnäs, Informative art: using amplified artworks as information displays, Proceedings of DARE 2000 on Designing augmented reality environments, p.103-114, April 2000, Elsinore, Denmark [doi>10.1145/354666.354677]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.1 MODELS AND PRINCIPLES
      H.1.2 User/Machine Systems
          Subjects: Human factors

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Evaluation/methodology

General Terms:
Design, Experimentation, Human Factors

Keywords:
empirical evaluation, monitoring, notification systems, peripheral displays