A model for notification systems evaluation—assessing user goals for multitasking activity

ABSTRACT

Addressing the need to tailor usability evaluation methods (UEMs) and promote effective reuse of HCI knowledge for computing activities undertaken in divided-attention situations, we present the foundations of a unifying model that can guide evaluation efforts for notification systems. Often implemented as ubiquitous systems or within a small portion of the traditional desktop, notification systems typically deliver information of interest in a parallel, multitasking approach, extraneous or supplemental to a user's attention priority. Such systems represent a difficult challenge to evaluate meaningfully. We introduce a design model of user goals based on blends of three critical parameters---interruption, reaction, and comprehension. Categorization possibilities form a logical, descriptive design space for notification systems, rooted in human information processing theory. This model allows conceptualization of distinct action models for at least eight classes of notification systems, which we describe and analyze with a human information processing model. System classification regions immediately suggest useful empirical and analytical evaluation metrics from related literature. We present a case study that demonstrates how these techniques can assist an evaluator in adapting traditional UEMs for notification and other multitasking systems. We explain why using the design model categorization scheme enabled us to generate evaluation results that are more relevant for the system redesign than the results of the original exploration done by the system's designers.

REFERENCES

Note: OCR errors may be found in this Reference List extracted from the full text article. ACM has opted to expose the complete List rather than only correct and linked references.

	1	Gregory D. Abowd , Elizabeth D. Mynatt, Charting past, present, and future research in ubiquitous computing, ACM Transactions on Computer-Human Interaction (TOCHI), v.7 n.1, p.29-58, March 2000 [doi>10.1145/344949.344988]
	2	Anderson, J. R. and Liebiere, C. 1998. The Atomic Components of Thought. Lawrence Erlbaum Associates, Inc., Mahwah, New Jersey.
	3	Baddelly, A. 1996. Exploring the central executive. The Quarterly Journal of Experimental Psychology 49A, 5--28.
	4	James A. Ballas , Constance L. Heitmeyer , Manuel A. Pérez-Quiñones, Evaluating two aspects of direct manipulation in advanced cockpits, Proceedings of the SIGCHI conference on Human factors in computing systems, p.127-134, May 03-07, 1992, Monterey, California, United States [doi>10.1145/142750.142770]
	5	Barnard, P. and May, J. 1999. Representing cognitive activity in complex tasks. Human-Computer Interaction 14, 93--158.
	6	Philip Barnard , Jon May , David Duke , David Duce, Systems, interactions, and macrotheory, ACM Transactions on Computer-Human Interaction (TOCHI), v.7 n.2, p.222-262, June 2000 [doi>10.1145/353485.353490]
	7	Bartram, L. 1998. Enhancing visualizations with motion. In Proceedings of the IEEE Symposium on Information Visualization (InfoVis '98). Raleigh, NC, 13--16.
	8	Bartram, L., Ware, C., and Calvert, T. 2001. Moving icons: Detection and distraction. In Proceedings of the IFIP TC.13 International Conference on Human-Computer Interaction (INTERACT 2001). Tokyo, Japan.
	9	Bartram, L. R. 2001. Enhancing information visualization with motion. Ph.D. thesis, Simon Fraser University, Canada.
	10	Cadiz, J., Venolia, G. D., Jancke, G., and Gupta, A. 2001. Sideshow: Providing peripheral awareness of important information. Tech. Rep. MSR-TR-2001-83, Microsoft Research, Collaboration, and Multimedia Group. Sept.
	11	C. M. Chewar , D. Scott McCrickard , Ali Ndiwalana , Chris North , Jon Pryor , David Tessendorf, Secondary task display attributes: optimizing visualizations for cognitive task suitability and interference avoidance, Proceedings of the symposium on Data Visualisation 2002, May 27-29, 2002, Barcelona, Spain
	12	Cutrell, E., Czerwinski, M., and Horvitz, E. 2001. Notification, disruption, and memory: Effects of messaging interruptions on memory and performance. In Proceedings of the IFIP TC.13 International Conference on Human-Computer Interaction (INTERACT 2001). Tokyo, Japan, 263--269.
	13	Andrew Dahley , Craig Wisneski , Hiroshi Ishii, Water lamp and pinwheels: ambient projection of digital information into architectural space, CHI 98 conference summary on Human factors in computing systems, p.269-270, April 18-23, 1998, Los Angeles, California, United States [doi>10.1145/286498.286750]
	14	Duchnicky, R. L. and Kolers, P. A. 1983. Readability of text scrolled on visual display terminals as a function of window size. Human Factors 25, 6, 683--692.
	15	Enns, J. T. and Rensink, R. A. 1991. Preattentive recovery of three-dimensional orientation from line drawings. Psychological Review 98, 335--351.
	16	Ericsson, K. A. and Kintsch, W. 1995. Long term working memory. Psychological Review 102, 2, 211--245.
	17	Foster, K. I. 1970. Visual perception of rapidly presented word sequences of varying complexity. Perception and Psychophysics 8, 215--221.
	18	Granaas, M. M., McKay, T. D., Laham, R. D., Hurt, L. D., and Juola, J. F. 1984. Reading moving text on a CRT screen. Human Factors 26, 1, 97--104.
	19	Gray, W. D. and Saltzman, M. C. 1998. Damaged merchandise? A review of experiments that compare usability evaluation methods. Human-Computer Interaction 13, 3, 203--261.
	20	Green, P. 1999. The 15-second rule for driver information systems. In Proceedings of the ITS America Ninth Annual Meeting. Washington, DC, CD--ROM.
	21	Saul Greenberg , Chester Fitchett, Phidgets: easy development of physical interfaces through physical widgets, Proceedings of the 14th annual ACM symposium on User interface software and technology, November 11-14, 2001, Orlando, Florida [doi>10.1145/502348.502388]
	22	Beverly L. Harrison , Hiroshi Ishii , Kim J. Vicente , William A. S. Buxton, Transparent layered user interfaces: an evaluation of a display design to enhance focused and divided attention, Proceedings of the SIGCHI conference on Human factors in computing systems, p.317-324, May 07-11, 1995, Denver, Colorado, United States [doi>10.1145/223904.223945]
	23	Christopher G. Healey , Kellogg S. Booth , James T. Enns, High-speed visual estimation using preattentive processing, ACM Transactions on Computer-Human Interaction (TOCHI), v.3 n.2, p.107-135, June 1996 [doi>10.1145/230562.230563]
	24	Christopher G. Healey , James T. Enns, Large Datasets at a Glance: Combining Textures and Colors in Scientific Visualization, IEEE Transactions on Visualization and Computer Graphics, v.5 n.2, p.145-167, April 1999 [doi>10.1109/2945.773807]
	25	Jeremy M. Heiner , Scott E. Hudson , Kenichiro Tanaka, The information percolator: ambient information display in a decorative object, Proceedings of the 12th annual ACM symposium on User interface software and technology, p.141-148, November 07-10, 1999, Asheville, North Carolina, United States [doi>10.1145/320719.322595]
	26	Eric Horvitz, Principles of mixed-initiative user interfaces, Proceedings of the SIGCHI conference on Human factors in computing systems: the CHI is the limit, p.159-166, May 15-20, 1999, Pittsburgh, Pennsylvania, United States [doi>10.1145/302979.303030]
	27	Horvitz, E., Jacobs, A., and Hovel, D. 1999. Attention-sensitive alerting. In Conference on Uncertainty and Artificial Intelligence (UAI '99). Stockholm, Sweden, 305--313.
	28	James M. Hudson , Jim Christensen , Wendy A. Kellogg , Thomas Erickson, "I'd be overwhelmed, but it's just one more thing to do": availability and interruption in research management, Proceedings of the SIGCHI conference on Human factors in computing systems: Changing our world, changing ourselves, April 20-25, 2002, Minneapolis, Minnesota, USA [doi>10.1145/503376.503394]
	29	Hiroshi Ishii , Craig Wisneski , Scott Brave , Andrew Dahley , Matt Gorbet , Brygg Ullmer , Paul Yarin, ambientROOM: integrating ambient media with architectural space, CHI 98 conference summary on Human factors in computing systems, p.173-174, April 18-23, 1998, Los Angeles, California, United States [doi>10.1145/286498.286652]
	30	Juola, J. F., Ward, N. J., and McNamara, T. 1982. Visual search and reading of rapid serial presentations of letter strings, words, and text. J. Exper. Psych. General 111, 2, 208--227.
	31	Kang, T. J. and Muter, P. 1989. Reading dynamically displayed text. Behaviour and Information Technology 8, 1, 33--42.
	32	Kieras, D. E. and Meyer, D. E. 1997. An overview of the EPIC architecture for cognition and performance with application to human-computer interaction. Human-Computer Interaction 12, 4, 391--438.
	33	Blair MacIntyre , Elizabeth D. Mynatt , Stephen Voida , Klaus M. Hansen , Joe Tullio , Gregory M. Corso, Support for multitasking and background awareness using interactive peripheral displays, Proceedings of the 14th annual ACM symposium on User interface software and technology, November 11-14, 2001, Orlando, Florida [doi>10.1145/502348.502355]
	34	Lena Mamykina , Elizabeth Mynatt , Michael A. Terry, Time Aura: interfaces for pacing, Proceedings of the SIGCHI conference on Human factors in computing systems, p.144-151, March 2001, Seattle, Washington, United States [doi>10.1145/365024.365077]
	35	McCrickard, D. S. 1999. Maintaining information awareness with Irwin. In Proceedings of the World Conference on Educational Multimedia/Hypermedia and Educational Telecommunications (ED-MEDIA '99). Seattle, WA.
	36	McCrickard, D. S., Catrambone, R., and Stasko, J. T. 2001. Evaluating animation in the periphery as a mechanism for maintaining awareness. In Proceedings of the IFIP TC.13 International Conference on Human-Computer Interaction (INTERACT 2001). Tokyo, Japan, 148--156.
	37	D. Scott McCrickard , C. M. Chewar, Attuning notification design to user goals and attention costs, Communications of the ACM, v.46 n.3, March 2003 [doi>10.1145/636772.636800]
	38	D. Scott McCrickard , Mary Czerwinski , Lyn Bartram, Introduction: design and evaluation of notification user interfaces, International Journal of Human-Computer Studies, v.58 n.5, p.509-514, May 2003 [doi>10.1016/S1071-5819(03)00025-9]
	39	Daniel Craig Mcfarlane , John Lee Sibert, Interruption of people in human-computer interaction, 1998
	40	McFarlane, D. C. 2002. Comparision of four primary methods for coordinating the interruption of people in human-computer interaction. Human Computer Interaction 17, 3.
	41	Allen Newell, Unified theories of cognition, Harvard University Press, Cambridge, MA, 1990
	42	William M. Newman, Better or just different? On the benefits of designing interactive systems in terms of critical parameters, Proceedings of the conference on Designing interactive systems: processes, practices, methods, and techniques, p.239-245, August 18-20, 1997, Amsterdam, The Netherlands [doi>10.1145/263552.263615]
	43	Norman, D. A. 1986. Cognitive engineering. In User Centered System Design: New Perspectives on Human Computer Interaction, D. A. Norman and S. W. Draper, Eds. Lawrence Erlbaum Associates, 31--62.
	44	Johan Redström , Tobias Skog , Lars Hallnäs, Informative art: using amplified artworks as information displays, Proceedings of DARE 2000 on Designing augmented reality environments, p.103-114, April 2000, Elsinore, Denmark [doi>10.1145/354666.354677]
	45	B. J. Rhodes , P. Maes, Just-in-time information retrieval agents, IBM Systems Journal, v.39 n.3-4, p.685-704, July 2000
	46	Rogers, R. D. and Monsell, S. 1995. Cost of a predictable switch between simple cognitive tasks. J. Exper. Psych. General 124, 2, 207--231.
	47	Mary Beth Rosson , John M. Carroll, Usability engineering: scenario-based development of human-computer interaction, Morgan Kaufmann Publishers Inc., San Francisco, CA, 2001
	48	See, J. E., Howe, S. R., Warm, J. S., and Dember, W. 1995. Meta analysis of the sensitivity decrement in vigilance. Psych. Bull. 117, 2, 230--249.
	49	Sheridan, T. 1991. Human factors of driver-vehicle interaction in the IVHS environment. Tech. Rep. DOT HS 807 837, National Technical Information Service, Springfield MA.
	50	Alistair Sutcliffe, On the effective use and reuse of HCI knowledge, ACM Transactions on Computer-Human Interaction (TOCHI), v.7 n.2, p.197-221, June 2000 [doi>10.1145/353485.353488]
	51	Tufano, D., Knee, H., and Spelt, P. 1996. In-vehicle signing functions and systems concepts. In Proceedings of the 29th International Symposium on Automotive Technology and Automation (ISATA) Dedicated Conference on Global Deployment of Advanced Transportation Telematics/ ITS. Florence Italy, 97--104.
	52	van Dantzich, M., Robbins, D., Horvitz, E., and Czerwinski, M. 2002. Scope: Providing awareness of multiple notifications at a glance. In Proceedings of the 6th International Working Conference on Advanced Visual Interfaces (AVI '02). ACM Press.
	53	Weiser, M. and Brown, J. S. 1996. Designing calm technology. PowerGrid Journal 1.01.
	54	Whittaker, S., Terveen, L., and Nardi, B. A. 2000. Let's stop pushing the envelope and start addressing it: A reference task agenda for HCI. Human-Computer Interaction 15, 75--106.
	55	Wickens, C. D. and Hollands, J. G. 2000. Engineering Psychology and Human Performance, Third ed. Prentice Hall, Upper Saddle River, NJ.