Evaluating information visualization applications with focus groups: the CourseVis experience

ABSTRACT

This paper reports our experience of evaluating an application that uses visualization approaches to support instructors in Web based distance education. The evaluation took place in three stages: a focus group, an experimental study, and a semi-structured interview. In this paper we focus our attention on the focus group, and we will show how this evaluation approach can be very effective in uncovering unexpected problems that cannot be identified with analytic evaluations or controlled experiments.

REFERENCES

• 1: Avgeriou, P., Papasalouros, A., Retalis, S., and Skordalakis, M. Towards a pattern language for learning management systems. Educational Technology & Society, 6(2): 11--24. http://ifets.ieee.org/periodical/6--2/2.html. 2003.
• 2: Stuart K. Card , Allen Newell , Thomas P. Moran, The Psychology of Human-Computer Interaction, Lawrence Erlbaum Associates, Inc., Mahwah, NJ, 1983
• 3: Chaomei Chen, Top 10 Unsolved Information Visualization Problems, IEEE Computer Graphics and Applications, v.25 n.4, p.12-16, July 2005 [doi>10.1109/MCG.2005.91]
• 4: Denzin, N. K. and Lincoln, Y. S. Collecting and interpreting qualitative materials. Sage, Thousand Oaks, CA. 1998.
• 5: Alan Dix , Janet Finley , Gregory Abowd , Russell Beale, Human-computer interaction (2nd ed.), Prentice-Hall, Inc., Upper Saddle River, NJ, 1998
• 6: Gibbs, A. Focus groups. Social Research Update, University of Surrey, UK. Num. 19. http://www.soc.surrey.ac.uk/sru/SRU19.html. 1997.
• 7: Greenbaum, T. L. The Handbook for Focus Group Research. SAGE publications, CA, USA. 1998
• 8: Johnson, P. Human-computer interaction: psychology, task analysis and software engineering. McGraw-Hill, London. 1992
• 9: Kaplan, B. and Maxwell, J. A. Qualitative research methods for evaluating computer information systems. In Anderson, J. G., Aydin, C., and Jay, S. J., editors, Valuating Health Care Information Systems: Methods and Applications, pages 45--68. Sage, Thousand Oaks, CA, 1994.
• 10: Krueger, R. A. Focus Groups: A Practical Guide for Applied Research. Third Edition. Sage Publishing, Newbury Park, CA. 2000.
• 11: Mazza R. Using Information Visualisation to Facilitate Instructors in Web-based Distance Learning. PhD thesis, University of Lugano, 2004.
• 12: Mazza R. and Milani C. Exploring Usage Analysis in Learning Systems: Gaining Insights From Visualisations. In: Workshop on Usage analysis in learning systems. 12th International Conference on Artificial Intelligence in Education (AIED 2005). Amsterdam, The Netherlands. 18 July 2005. pp 65--72. 2005.
• 13: Riccardo Mazza , Vania Dimitrova, Generation of Graphical Representations of Student Tracking Data in Course Management Systems, Proceedings of the Ninth International Conference on Information Visualisation (IV'05), p.253-258, July 06-08, 2005 [doi>10.1109/IV.2005.54]
• 14: Riccardo Mazza , Vania Dimitrova, Visualising student tracking data to support instructors in web-based distance education, Proceedings of the 13th international World Wide Web conference on Alternate track papers & posters, May 19-21, 2004, New York, NY, USA [doi>10.1145/1013367.1013393]
• 15: Morgan, D. L. Focus group as qualitative research. Sage, London. 1988.
• 16: Jakob Nielsen, The Use and Misuse of Focus Groups, IEEE Software, v.14 n.1, p.94-95, January 1997 [doi>10.1109/52.566434]
• 17: Catherine Plaisant, The challenge of information visualization evaluation, Proceedings of the working conference on Advanced visual interfaces, May 25-28, 2004, Gallipoli, Italy [doi>10.1145/989863.989880]
• 18: Melanie Tory , Torsten Möller, Human Factors in Visualization Research, IEEE Transactions on Visualization and Computer Graphics, v.10 n.1, p.72-84, January 2004 [doi>10.1109/TVCG.2004.1260759]
• 19: Web CT. World Wide Web course tool. WebCT educational technologies, Vancouver, Canada. http://www.webct.com.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Evaluation/methodology

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Graphical user interfaces (GUI)

General Terms:
Human Factors, Verification

Keywords:
focus group, human factors, information visualization evaluation