Modeling and understanding students' off-task behavior in intelligent tutoring systems

ABSTRACT

We present a machine-learned model that can automatically detect when a student using an intelligent tutoring system is off-task, i.e., engaged in behavior which does not involve the system or a learning task. This model was developed using only log files of system usage (i.e. no screen capture or audio/video data). We show that this model can both accurately identify each student's prevalence of off-task behavior and can distinguish off-task behavior from when the student is talking to the teacher or another student about the subject matter. We use this model in combination with motivational and attitudinal instruments, developing a profile of the attitudes and motivations associated with off-task behavior, and compare this profile to the attitudes and motivations associated with other behaviors in intelligent tutoring systems. We discuss how the model of off-task behavior can be used within interactive learning environments which respond to when students are off-task.

REFERENCES

• 1: Aleven, V., McLaren, B.M., Roll, I., and Koedinger, K.R. Toward tutoring help seeking: Applying cognitive modeling to meta-cognitive skills. Proceedings of the 7th International Conference on Intelligent Tutoring Systems (ITS 2004), 227--239.
• 2: Anderson, J.R., Corbett, A.T., Koedinger, K.R., Pelletier, R. Cognitive Tutors: Lessons Learned. Journal of the Learning Sciences 4, 2 (1995), 167--207.
• 3: Amershi, S., and Conati, C. Automatic Recognition of Learner Groups in Exploratory Learning Environments. Proceedings of the 8th International Conference on Intelligent Tutoring Systems (ITS 2006), 463--472.
• 4: Ryan Shaun Baker , Albert T. Corbett , Kenneth R. Koedinger, Designing intelligent tutors that adapt to when students game the system, Carnegie Mellon University, Pittsburgh, PA, 2006
• 5: Baker, R.S., Corbett, A.T., and Koedinger, K.R. Detecting Student Misuse of Intelligent Tutoring Systems. Proceedings of the 7th International Conference on Intelligent Tutoring Systems (ITS 2004), 531--540.
• 6: Baker, R.S.J.d., Corbett, A.T., Roll, I., Wagner, A.Z., and Koedinger, K.R. The Relationship Between Gaming the System and Learning in Cognitive Tutor Classrooms. Manuscript Under Review.
• 7: Ryan Shaun Baker , Albert T. Corbett , Kenneth R. Koedinger , Angela Z. Wagner, Off-task behavior in the cognitive tutor classroom: when students "game the system", Proceedings of the SIGCHI conference on Human factors in computing systems, p.383-390, April 24-29, 2004, Vienna, Austria [doi>10.1145/985692.985741]
• 8: Baker, R.S., Roll, I., Corbett, A.T., Koedinger, K.R. Do Performance Goals Lead Students to Game the System? Proceedings of the 12th International Conference on Artificial Intelligence in Education (AIED 2005), 57--64.
• 9: Beck, J.E. Engagement tracing: using response times to model student disengagement. Proceedings of the 12th International Conference on Artificial Intelligence in Education (AIED 2005), 88--95.
• 10: Timothy W. Bickmore , Rosalind W. Picard, Towards caring machines, CHI '04 extended abstracts on Human factors in computing systems, April 24-29, 2004, Vienna, Austria [doi>10.1145/985921.986097]
• 11: Stephen Boyd , Lieven Vandenberghe, Convex Optimization, Cambridge University Press, New York, NY, 2004
• 12: Carroll, J. A Model For School Learning. Teachers College Record 64 (1963), 723--733.
• 13: Cohen, J. A coefficient of agreement for nominal scales. Educational and Psychological Measurement 20, 1 (1960), 37--46.
• 14: Collins, R.T., Lipton, A.J., Kanade, T., Fujiyoshi, H., Duggins, D., Tsin, Y., Tolliver, D., Enomoto, N., Hasegawa, O., Burt, P., Wixson, L. Carnegie Mellon University Technical Report CMU-RI-TR-00-12: A System for Video Surveillance and Monitoring (2000).
• 15: Dalton, T., Martella, R.C., and Marchand-Martella, N.E. The effects of a self-management program in reducing off-task behavior. Journal of Behavioral Education 9, 3--4 (1999), 157--176.
• 16: Angel de Vicente , Helen Pain, Informing the Detection of the Students' Motivational State: An Empirical Study, Proceedings of the 6th International Conference on Intelligent Tutoring Systems, p.933-943, June 02-07, 2002
• 17: Dillon, T.W., Garner, M., Kuilboer, J., Quinn, J.D. Accounting Student Acceptance of Tax Preparation Software. Journal of Accounting and Computers 13 (1998), 17--29.
• 18: Cristina Conati , Stacy Marsella , Ana Paiva, Affective interactions: the computer in the affective loop, Proceedings of the 10th international conference on Intelligent user interfaces, January 10-13, 2005, San Diego, California, USA [doi>10.1145/1040830.1040838]
• 19: Frantom, C.G., Green, K.E., Hoffman, E.R. Measure Development: The Children's Attitudes Towards Technology Scale (CATS). Journal of Educational Computing Research 26, 3 (2002), 249--263.
• 20: Harnisch, D.L., Hill, K.T., Fyans, L.J. Development of a Shorter, More Reliable, and More Valid Measure of Test Motivation. Paper presented at the 1980 annual meeting of the National Council on Measurement in Education. ERIC Document #ED193273.
• 21: Knezek, G., Christensen, R. Computer Attitudes Questionnaire (1995). Denton, TX: Texas Center for Educational Technology.
• 22: Maris, E. Psychometric Latent Response Models. Psychometrika 60, 4 (1995), 523--547.
• 23: Mueller, C.M., and Dweck, C.S. Praise for Intelligence Can Undermine Children's Motivation and Performance. Journal of Personality and Social Psychology 75, 1 (1998), 33--52.
• 24: Murray, R.C., and vanLehn, K. Effects of Dissuading Unnecessary Help Requests While Providing Proactive Help. Proceedings of the 12th International Conference on Artificial Intelligence in Education (AIED 2005), 887--889.
• 25: Parker, G., Hadzi-Pavlovic, D. A Question of Style: Refining the Dimensions of Personality Disorder Style. Journal of Personality Disorders, 15, 4 (2001), 300--318.
• 26: Ramsey, F.L., Schafer, D.W. The Statistical Sleuth: A Course in Methods of Data Analysis. Duxbury Press, Belmont, CA, USA, 1997.
• 27: Sarason, S.B. Anxiety in Elementary School Children: A Report of Research. Greenwood Press, Westport, CT, USA, 1978.
• 28: Neil Selwyn, Students' attitudes toward computers: validation of a computer attitude scale for 16--19 education, Computers & Education, v.28 n.1, p.35-41, Jan. 1997 [doi>10.1016/S0360-1315(96)00035-8]
• 29: Lucy A. Suchman, Plans and situated actions: the problem of human-machine communication, Cambridge University Press, New York, NY, 1987
• 30: Yu, L., and Liu, H. Feature selection for high-dimensional data: a fast correlation-based filter solution. Proceedings of the International Conference on Machine Learning (2003), 856--863.
• 31: Tina R. Ziemek, Two-D or not Two-D: gender implications of visual cognition in electronic games, Proceedings of the 2006 symposium on Interactive 3D graphics and games, March 14-17, 2006, Redwood City, California [doi>10.1145/1111411.1111444]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.m Miscellaneous

Additional Classification:
  K. Computing Milieux
  K.3 COMPUTERS AND EDUCATION
      K.3.m Miscellaneous

General Terms:
Human Factors

Keywords:
intelligent tutoring systems, motivation, off-task behavior, user attitudes, user modeling