Evaluating user and system models: applying scaling techniques to problems in human-computer interaction

ABSTRACT

A user's mental model of a system should be an important determinant of performance and as well as a means of understanding why particular user errors occur. In particular, experienced users' models should be in closer agreement with the system than less experienced users' models, and deviations of expert models from the system should correspond to difficulties in performance and suggest ways that system usability could be improved. The present study explored the utility of scaling techniques for defining and comparing user and system models. The results support the assertion that with experience users' mental models approach the system model. However, even experienced users had significant deviations from the system model, leading to predictions of where experts would have difficulty using the system and suggestions for improving usability.

REFERENCES

• 1: Adelson, B. When novices surpass experts: The difficulty of a task can increase with expertise. Journal Experimental Psychology: Learning, Memory and Cognition, 10(3), 1984, 483-495.
• 2: John R. Anderson, The Architecture of Cognition, Harvard University Press, Cambridge, MA, 1983
• 3: Baggett, P., and Ehrenfeucht, A. Conceptualizing assembly tasks. Technical Report No. 139, Institute of Cognitive Science, University of Colorado, Boulder, Colorado 80309, 1985.
• 4: Bonar, J., and Soloway, E. Preprogramming knowledge: A major source of misconceptions in novice programmers. Human-Computer Interaction, 1(2), 1986, 133-162.
• 5: John M. Carroll , Robert L. Mack, Learning to use a word processor: by doing, by thinking, and by knowing, Human factors in computer systems, Ablex Publishing Corp., Norwood, NJ, 1984
• 6: Cohen, J., and Cohen, P. Applied Multiple Regression~Correlation Analysis for the Behavioral Sciences. Erlbaum: Hillsdale, New Jersey, 1975.
• 7: Cooke, N.M., and Schvaneveldt, R.W. Variations in cognitive networks with computer programming experience. Unpublished manuscript, New Mexico State University, Las Cruces, New Mexico 88003, 1986.
• 8: Hays, W.L. Statistics for the Social Sciences, 2nd Edition. Holt, Rinehart and Winston, Inc.: New York, New York, 1973.
• 9: Kay, D.S., and Black, J.B. Changes in knowledge representations of computer systems with experience. Proceedings of the Human Factors Society, 28th Annual Meeting, San Antonio, Texas, 1984.
• 10: Kay, D.S., and Black, J.B. The evolution of knowledge representations with increasing expertise in using systems. Proceedings of the Seventh Annual Conference of the Cognitive Science Society, Irvine, California, 1985.
• 11: Kellogg, W.A., and Breen, T.J. Using psychological scaling techniques to evaluate user and system models. Unpublished manuscript, T.J. Watson Research Center, Yorktown Heights, NY, 1987.
• 12: Schvaneveldt, R.W., Durso, F.T., and Dearholt, D.W. Pathfinder: Scaling with network structures. Memoranda in Computer and Cognitive Science, Report No. MCCS-85-9, Computer Research Laboratory, New Mexico State University, 1985.
• 13: Sehvaneveldt, R.W., Durso, F.T., Goldsmith, T.E., Breen, T.J., Cooke, N.J., Tucker, R.G., and DeMaio, J.C. Measuring the structure of expertise. Memoranda in Computer and Cognitive Science, Report No. MCCS-85-32, Computer Research Laboratory, New Mexico State University, 1985.
• 14: Sebrechts, M.M., Black, J.B., Galambos, J.A., Wagner, R.K., Deck, J.A., and Wikler, E.A. The effects of diagrams on learning to use a system. Report No. 2, Learning and Using Systems, Yale University, New Haven, Connecticut, 1983.
• 15: Stevens, A., Collins, A., and Goldin, S. Misconceptions in students' understanding. International Journal of Man-Machine Studies, 11, 1979, 145-156.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Evaluation/methodology

Additional Classification:
  H. Information Systems
  H.1 MODELS AND PRINCIPLES

  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.0 General
          Subjects: Cognitive simulation

General Terms:
Design, Measurement, Performance, Reliability, Theory

Keywords:
assessment of system usability, mental models, psychological scaling techniques, user conceptual models