Exploring the applications of user-expertise assessment for intelligent interfaces

ABSTRACT

An adaptive user interface relies, to a large extent, upon an adequate user model (e.g., a representation of user-expertise). However, building a user model may be a tedious and time consuming task that will render such an interface unattractive to developers. We thus need an effective means of inferring the user model at low cost. In this paper, we describe a technique for automatically inferring a fine-grain model of a user's knowledge state based on a small number of observations. With this approach, the domain of knowledge to be evaluated is represented as a network of nodes (knowledge units—KU) and links (implications) induced from empirical user profiles. The user knowledge state is specified as a set of weights attached to the knowledge units that indicate the likelihood of mastery. These weights are updated every time a knowledge unit is reassigned a new weight (e.g., by a question-and-answer process). The updating scheme is based on the Dempster-Shafer algorithm. A User Knowledge Assessment Tool (UKAT) that employs this technique has been implemented. By way of simulations we explore an entropy-based method of choosing questions, and compare the results with a random sampling method. The experimental results show that the proposed knowledge assessment and questioning methods are useful and efficient in inferring detailed models of user-expertise, but the entropy-based method can induce a bias in some circumstances.

REFERENCES

• 1: F. de Rosis, S. Pizzutilo, A. Russo, D. C. Berry, and E J. Nicolau Molina. Modeling the user knowledge by belief networks. User Modeling and User-Adapted Interaction, 2(4), 1992.
• 2: Michel C. Desmarais. Architecture etfondements empiriques d'un systdme d'aide assist~e par ordinateur pour r ~dition de texte. PhD thesis, Universit6 de Mont6al, D6partement de psychologie, 1990.
• 3: Michel C. Desmarais, Luc Giroux, and Serge Larochelle. Fondements m6thodologiques et empiriques d'un syst~me consultant actif pour l'6dition de texte ~ le projet EdCoach. Technologies de l' information et societY, 4(1):61-74,1992.
• 4: Michel C. Desmarais, Luc Giroux, Serge Larochelle, and Serge Leclerc. Assessing the structure of knowledge in a procedural domain. In Proceedings of the Cognitive Science Society, pages 475-481, 14-17 August, Montr6al 1988.
• 5: Michel C. Desmarais , Jiming Liu, Experimental results on user knowledge assessment with an evidential reasoning methodology, Proceedings of the 1st international conference on Intelligent user interfaces, p.223-226, January 04-07, 1993, Orlando, Florida, United States [doi>10.1145/169891.169994]
• 6: Jean-Claude Falmagne and Jean-Paul Doignon. A class of stochastic procedures for the assessment of knowledge. British Journal of Mathematical and Statistical Psychology, 41" 1-23, 1988.
• 7: Jean-Claude Falmagne, Jean-Paul Doignon, Mathieu Koppen, Michael Villano, and Leila Johannesen. Introduction to knowledge spaces: how to build, test and search them. Psychological Review, 97(2):201-224, 1990.
• 8: I.P. Goldstein. The genetic graph: a representation for the evolution of procedural knowledge. In Sleeman and Brown {15}, pages 51-77.
• 9: J. Gordon and E. H. Shortliffe. The dempster-shafer theory of evidence. In B. G. Buchanan and E. H. Shortliffe, editors, Rule-Based Expert Systems. Addison-Wesley, Reading, M. A., 1984.
• 10: Alfred Kobsa, Modeling the user's conceptual knowledge in BGP-MS, a user modeling shell system, Computational Intelligence, v.6 n.4, p.193-208, Nov. 1990 [doi>10.1111/j.1467-8640.1990.tb00295.x]
• 11: Jiming Liu and Michel C. Desmarais. Knowledge assessment based on the dempster-shafer belief propagation theory. Technical Report CRIM-92/09--06, Centre de recherche informatique de Montr6al, 1992.
• 12: J. Sell Student models: what use are they? In P. Ercoli and R. Lewis, editors, Artificial intelligence tools in education. North-Holland, Amsterdam, 1988.
• 13: G. Sharer. A Mathematical Theory of Evidence. Princeton University Press, Princeton, N. J., 1976.
• 14: D. Sleeman, UMFE: a user modelling front-end subsystem, International Journal of Man-Machine Studies, v.23 n.1, p.71-88, July 1985 [doi>10.1016/S0020-7373(85)80025-0]
• 15: D. Sleeman andJ.S. Brown, editors.lntelligent Tutoring Systems. Academic Press, London, 1982.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Evaluation/methodology

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Theory and methods

  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.4 Knowledge Representation Formalisms and Methods
          Subjects: Representations (procedural and rule-based)

General Terms:
Algorithms, Experimentation, Human Factors, Performance, Theory

Keywords:
adaptive training systems, entropy, evidence aggregation, intelligent interfaces, knowledge spaces, probabilistic reasoning, user-expertise assessment