Evaluating information visualisations

ABSTRACT

As more experience is being gained with the evaluation of information visualisation interfaces, weaknesses in current evaluation practice are coming to the fore.This position paper presents an overview of currently used evaluation methods, followed by a discussion of my experiences and lessons learned from a series of studies comparing hierarchy browsers.

REFERENCES

• 1: K. Andrews. Human-Computer Interaction: Lecture Notes. 2006. http://courses.iicm.edu/hci/hci.pdf.
• 2: Keith Andrews , Wolfgang Kienreich , Vedran Sabol , Jutta Becker , Georg Droschl , Frank Kappe , Michael Granitzer , Peter Auer , Klaus Tochtermann, The InfoSky visual explorer: exploiting hierarchical structure and document similarities, Information Visualization, v.1 n.3/4, p.166-181, December 2002 [doi>10.1057/palgrave.ivs.9500023]
• 3: M. D. Byrne, B. E. John, and E. Joyce. A Day in the Life of Ten WWW Users, 2000. http://citeseer.ist.psu.edu/400156.html. Unpublished paper.
• 4: Michael D. Byrne , Bonnie E. John , Neil S. Wehrle , David C. Crow, The tangled Web we wove: a taskonomy of WWW use, Proceedings of the SIGCHI conference on Human factors in computing systems: the CHI is the limit, p.544-551, May 15-20, 1999, Pittsburgh, Pennsylvania, United States [doi>10.1145/302979.303154]
• 5: I. Ceaparu, J. Lazar, K. Bessiere, J. Robinson, and B. Shneiderman. Determining Causes and Severity of End-User Frustration. Technical Report CS-TR-4371, University of Maryland, Computer Science Department, May 2003. http://hcil.cs.umd.edu/trs/2002--11/2002--11.pdf. Revised Version.
• 6: Chaomei Chen , Yue Yu, Empirical studies of information visualization: a meta-analysis, International Journal of Human-Computer Studies, v.53 n.5, p.851-866, Nov. 2000 [doi>10.1006/ijhc.2000.0422]
• 7: K. A. Ericsson and H. A. Simon. Protocol Analysis: Verbal Reports As Data. MIT Press, revised edition, May 1993. ISBN 0262550237.
• 8: J.-D. Fekete and C. Plaisant. Information Visualization Benchmarks Repository. 2003. http://www.cs.umd.edu/hcil/InfovisRepository/contest-2003/.
• 9: A. Field. Discovering Statistics Using SPSS for Windows. Sage Publications, 2000. ISBN 0761957553.
• 10: A. Field and G. Hole. How to Design and Report Experiments. Sage Publications, 2002. ISBN 0761973834.
• 11: Michael Granitzer , Wolfgang Kienreich , Vedran Sabol , Keith Andrews , Werner Klieber, Evaluating a System for Interactive Exploration of Large, Hierarchically Structured Document Repositories, Proceedings of the IEEE Symposium on Information Visualization (INFOVIS'04), p.127-134, October 10-12, 2004 [doi>10.1109/INFOVIS.2004.19]
• 12: Elizabeth Hetzler , Alan Turner, Analysis Experiences Using Information Visualization, IEEE Computer Graphics and Applications, v.24 n.5, p.22-26, September 2004 [doi>10.1109/MCG.2004.22]
• 13: David M. Hilbert , David F. Redmiles, Extracting usability information from user interface events, ACM Computing Surveys (CSUR), v.32 n.4, p.384-421, Dec. 2000 [doi>10.1145/371578.371593]
• 14: Alfred Kobsa, User Experiments with Tree Visualization Systems, Proceedings of the IEEE Symposium on Information Visualization (INFOVIS'04), p.9-16, October 10-12, 2004 [doi>10.1109/INFOVIS.2004.70]
• 15: J. Kort. CHI2005 Workshop 6 Results. April 2005. http://usage.n1/workshop.html.
• 16: Joke Kort , Henk de Poot, Usage analysis: combining logging and qualitative methods, CHI '05 extended abstracts on Human factors in computing systems, April 02-07, 2005, Portland, OR, USA [doi>10.1145/1056808.1057117]
• 17: T. Landauer. Research Methods in HCI. In M. G. Helander, editor, Handbook of Human-Computer Interaction, chapter 42, pages 905--928. North-Holland, 1988. ISBN 0444705368.
• 18: G. Modes. A Usage Study of Desktop Users: Technical Realisation and Coding Software. Master's thesis, Graz University of Technology, Austria, September 2005.
• 19: Catherine Plaisant, The challenge of information visualization evaluation, Proceedings of the working conference on Advanced visual interfaces, May 25-28, 2004, Gallipoli, Italy [doi>10.1145/989863.989880]
• 20: W. Putz. The Hierarchical Visualization System: A General Framework for Visualizing Information Hierarchies Using the Example of Information Pyramids. Master's thesis, Graz University of Technology, Austria, March 2005. ftp://ftp.iicm.edu/pub/theses/wputz.pdf.
• 21: Abigail J. Sellen , Rachel Murphy , Kate L. Shaw, How knowledge workers use the web, Proceedings of the SIGCHI conference on Human factors in computing systems: Changing our world, changing ourselves, April 20-25, 2002, Minneapolis, Minnesota, USA [doi>10.1145/503376.503418]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Evaluation/methodology

Additional Classification:
  H. Information Systems
  H.1 MODELS AND PRINCIPLES
      H.1.2 User/Machine Systems
          Subjects: Human factors

  I. Computing Methodologies
  I.6 SIMULATION AND MODELING
      I.6.8 Types of Simulation
          Subjects: Visual

General Terms:
Design, Measurement, Performance