The challenge of information visualization evaluation

ABSTRACT

As the field of information visualization matures, the tools and ideas described in our research publications are reaching users. The reports of usability studies and controlled experiments are helpful to understand the potential and limitations of our tools, but we need to consider other evaluation approaches that take into account the long exploratory nature of users tasks, the value of potential discoveries or the benefits of overall awareness. We need better metrics and benchmark repositories to compare tools, and we should also seek reports of successful adoption and demonstrated utility.

REFERENCES

• 1: Borzo, Janet, Get the picture, Wall Street Journal, January 12 (2004).
• 2: National Cancer Institute Cancer Mortality Maps and Graphs www3.cancer.gov/atlasplus.
• 3: Sytadin. www.sytadin.tm.fr.
• 4: Komlodi, A., Sears, A., Stanziola, E., Information Visualization Evaluation Review, ISRC Tech. Report, Dept. of Information Systems, UMBC. UMBC-ISRC-2004-1 http://www.research.umbc.edu/~komlodi/IV_eval (2004).
• 5: Christopher Ahlberg , Ben Shneiderman, The alphaslider: a compact and rapid selector, Proceedings of the SIGCHI conference on Human factors in computing systems: celebrating interdependence, p.365-371, April 24-28, 1994, Boston, Massachusetts, United States [doi>10.1145/191666.191790]
• 6: Pourang Irani , Colin Ware, Diagramming information structures using 3D perceptual primitives, ACM Transactions on Computer-Human Interaction (TOCHI), v.10 n.1, p.1-19, March 2003 [doi>10.1145/606658.606659]
• 7: A. G. Sutcliffe , M. Ennis , J. Hu, Evaluating the effectiveness of visual user interfaces for information retrieval, International Journal of Human-Computer Studies, v.53 n.5, p.741-763, Nov. 2000 [doi>10.1006/ijhc.2000.0416]
• 8: Donald Byrd, A scrollbar-based visualization for document navigation, Proceedings of the fourth ACM conference on Digital libraries, p.122-129, August 11-14, 1999, Berkeley, California, United States [doi>10.1145/313238.313283]
• 9: Catherine Plaisant , Jesse Grosjean , Benjamin B. Bederson, SpaceTree: Supporting Exploration in Large Node Link Tree, Design Evolution and Empirical Evaluation, Proceedings of the IEEE Symposium on Information Visualization (InfoVis'02), p.57, October 28-29, 2002
• 10: J. Gregory Trafton , Susan S. Kirschenbaum , Ted L. Tsu , Robert T. Miyamoto , James A. Ballas , Paula D. Raymond, Turning pictures into numbers: extracting and generating information from complex visualizations, International Journal of Human-Computer Studies, v.53 n.5, p.827-850, Nov. 2000 [doi>10.1006/ijhc.2000.0419]
• 11: Moore, G., Crossing the Chasm, HarpersCollins Publishers, New York, NY (1991).
• 12: Moore, G., Inside the Tornado, HarpersCollins Publishers, New York, NY (1995).
• 13: Chaomei Chen , Mary P. Czerwinski, Empirical evaluation of information visualizations: an introduction, International Journal of Human-Computer Studies, v.53 n.5, p.631-635, Nov. 2000 [doi>10.1006/ijhc.2000.0421]
• 14: Plaisant, C., Information visualization and the challenge of universal access, in Exploring Geovisualization, Dykes, J., MacEachren, A. and Kraak, M.-J. (Eds.), Elsevier, Oxford (2004) in press.
• 15: Zhao, H., Shneiderman, B., Image-based highly interactive web mapping for geo-referenced data publishing, Tech Report CS-TR-4431. (2003) www.cs.umd.edu/hcil.
• 16: Stephen Brewster, Nonspeech auditory output, The human-computer interaction handbook: fundamentals, evolving technologies and emerging applications, Lawrence Erlbaum Associates, Inc., Mahwah, NJ, 2002
• 17: Zhao, H., Plaisant, C., Shneiderman, B., Duraiswani, R., Sonification of geo-referenced data for auditory information seeking: Design principle and pilot study, University of Maryland Tech. Report HCIL 2004-03 (2003) available at www.cs.umd.edu/hcil.
• 18: Brewer, Cynthia, ColorBrewer, www.personal.psu.edu/faculty/c/a/cab38/ColorBrewerBeta.html.
• 19: Dougherty, R., Wade, A. VisCheck: www.vischeck.com
• 20: Ben Shneiderman, Promoting universal usability with multi-layer interface design, Proceedings of the 2003 conference on Universal usability, p.1-8, November 10-11, 2003, Vancouver, British Columbia, Canada
• 21: Fekete, J-D and Plaisant, C., InfoVis 2003 Contest, www.cs.umd.edu/hcil/iv03contest (2003)
• 22: Fekete, J.-D., Greinstein, G. and Plaisant, C., InfoVis 2004 Contest, www.cs.umd.edu/hcil/iv04contest
• 23: Fekete, J-D and Plaisant, C., Information Visualization Benchmark Repository www.cs.umd.edu/hcil/InfovisRepository
• 24: Tamara Munzner , François Guimbretière , Serdar Tasiran , Li Zhang , Yunhong Zhou, TreeJuxtaposer: scalable tree comparison using Focus+Context with guaranteed visibility, ACM Transactions on Graphics (TOG), v.22 n.3, July 2003
• 25: Hong, J. Y., D'Andries, J., Richman, M., Westfall, M., Zoomology: Comparing Two Large Hierarchical Trees, in Poster Compendium of IEEE Information Visualization (2003) (also available at www.cs.umd.edu/hcil/iv03contest)
• 26: Spenke, M., Beilken, C., InfoZoom - Analysing Formula One racing results with an interactive data mining and visualization tool, Ebecken, N. Data mining II, (2000), 455--464.
• 27: Auber, D., Delest, M., Domenger, J-P., Ferraro, P., Strandh, R., EVAT - Environment for Visualization and Analysis of Trees, in Poster Compendium of IEEE Information Visualization (2003) (also available at www.cs.umd.edu/hcil/iv03contest).
• 28: Morse, D. R., Ytow, N., Roberts, D. McL., Sato, A., Comparison of Multiple Taxonomic Hierarchies Using TaxoNote, in Poster Compendium of IEEE Information Visualization (2003) (also available at www.cs.umd.edu/hcil/iv03contest).
• 29: Sheth, N., Börner, K., Baumgartner, J., Mane, K., Wernert, E., Treemap, Radial Tree, and 3D Tree Visualizations, in Poster Compendium of IEEE Information Visualization (2003) (also available at www.cs.umd.edu/hcil/iv03contest)
• 30: Alexander, Dey, Return on investment resources, http://deyalexander.com/resources/roi.html.
• 31: Dürsteler, Juan Carlos, www.infovis.net.
• 32: Christopher Ahlberg , Ben Shneiderman, Visual information seeking: tight coupling of dynamic query filters with starfield displays, Proceedings of the SIGCHI conference on Human factors in computing systems: celebrating interdependence, p.313-317, April 24-28, 1994, Boston, Massachusetts, United States [doi>10.1145/191666.191775]
• 33: Shneiderman, B. Dynamic queries, starfield displays, and the path to Spotfire, www.cs.umd.edu/hcil/spotfire.
• 34: Shneiderman, B., Treemaps for space-constrained visualization of hierarchies, www.cs.umd.edu/hcil/treemaps
• 35: Ben Shneiderman, Tree visualization with tree-maps: 2-d space-filling approach, ACM Transactions on Graphics (TOG), v.11 n.1, p.92-99, Jan. 1992 [doi>10.1145/102377.115768]
• 36: Benjamin B. Bederson , Ben Shneiderman , Martin Wattenberg, Ordered and quantum treemaps: Making effective use of 2D space to display hierarchies, ACM Transactions on Graphics (TOG), v.21 n.4, p.833-854, October 2002 [doi>10.1145/571647.571649]
• 37: Plaisant, C., Facilitating Data Exploration: Dynamic Queries on a Health Statistics Map. Proc. of the Annual Meeting of the American Statistical Association (Government Statistics Section), ASA, Alexandria, VA (1993) 18--23.
• 38: Li, Q., North, C., Empirical comparison of dynamic queries sliders and brushing histograms, Proc. IEEE Information Visualization (2003), 147--15.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)

General Terms:
Human Factors, Measurement, Performance

Keywords:
adoption, evaluation, return on investment, technology transfer, usability, usefulness, user studies, visualization