Lessons learned in modeling schizophrenic and depressed responsive virtual humans for training

ABSTRACT

This paper describes lessons learned in developing the linguistic, cognitive, emotional, and gestural models underlying virtual human behavior in a training application designed to train civilian police officers how to recognize gestures and verbal cues indicating different forms of mental illness and how to verbally interact with the mentally ill. Schizophrenia, paranoia, and depression were all modeled for the application. For linguistics, the application has quite complex language grammars that captured a range of syntactic structures and semantic categories. For cognition, there is a great deal of augmentation to a plan-based transition network needed to model the virtual humans knowledge. For emotions and gestures, virtual human behavior is based on expert-validated mapping tables specific to each mental illness. The paper presents five areas demanding continued research to improve virtual human behavior for use in training applications

REFERENCES

• 1: André, E., Klesen, M., Gebhard, P., Allen, S., & Rist, T. (2000). Exploiting Models of Personality and Emotions to Control the Behavior of Animated Interface Agents. In J. Rickel et al. (Eds.), Proceedings of the Fourth International Conference on Autonomous Agents (pp. 3--7), Barcelona, June 2000.
• 2: André, E., Rist, T., & Müüller, J. (1999). Employing AI Methods to Control the Behavior of Animated Interface Agents. International Journal of Applied Artificial Intelligence, 13(4-5), 415--448.
• 3: Bécheiraz, P., & Thalmann, D. (1998). A Behavioral Animation System for Autonomous Actors Personified by Emotions. Proceedings of the First Workshop on Embodied Conversational Characters, Lake Tahoe, CA.
• 4: Berenbaum, H., & Oltmanns, T.F. (1992). Emotional Experience and Expression in Schizophrenia and Depression. Journal of Abnormal Psychology, 101(1), 37--44.
• 5: J. Cassell , T. Bickmore , H. Vilhjálmsson , H. Yan, More than just a pretty face: affordances of embodiment, Proceedings of the 5th international conference on Intelligent user interfaces, p.52-59, January 09-12, 2000, New Orleans, Louisiana, United States [doi>10.1145/325737.325781]
• 6: Justine Cassell , Catherine Pelachaud , Norman Badler , Mark Steedman , Brett Achorn , Tripp Becket , Brett Douville , Scott Prevost , Matthew Stone, Animated conversation: rule-based generation of facial expression, gesture & spoken intonation for multiple conversational agents, Proceedings of the 21st annual conference on Computer graphics and interactive techniques, p.413-420, July 1994 [doi>10.1145/192161.192272]
• 7: Cassell, J., & Thóórisson, K.R. (1999). The Power of a Nod and a Glance: Envelope vs. Emotional Feedback in Animated Conversational Agents. International Journal of Applied Artificial Intelligence, 13(4-5), 519--538.
• 8: J. Cassell , H. Vilhjálmsson, Fully Embodied Conversational Avatars: Making Communicative Behaviors Autonomous, Autonomous Agents and Multi-Agent Systems, v.2 n.1, p.45-64, March 1999 [doi>10.1023/A:1010027123541]
• 9: Chovil, N. (1992). Discourse-Oriented Facial Displays in Conversation. Research on Language and Social Interaction, 25, 163--194.
• 10: Elliott, C. (1993). Using the Affective Reasoner to Support Social Simulations. In Proceedings of the Thirteenth International Joint Conference on Artificial Intelligence (pp. 194--200), Chambery, France, August 1993. Morgan Kaufmann.
• 11: Fehr, B., & Russell, J.A. (1984). Concept of Emotion Viewed From a Prototype Perspective. Journal of Experimental Psychology: General, 113(3), 464-486.
• 12: Frank, G.A., Guinn, C.I., Hubal, R.C., Stanford, M.A., Pope, P., & Lamm-Weisel, D. (2002). JUST-TALK: An Application of Responsive Virtual Human Technology. In Proceedings of I/ITSEC, Orlando, FL.
• 13: John Funge , Xiaoyuan Tu , Demetri Terzopoulos, Cognitive modeling: knowledge, reasoning and planning for intelligent characters, Proceedings of the 26th annual conference on Computer graphics and interactive techniques, p.29-38, July 1999 [doi>10.1145/311535.311538]
• 14: Groves, R., & Couper, M. (1998). Nonresponse in Household Interview Surveys. New York: John Wiley & Sons, Inc.
• 15: Gunther-Mohr, C. (1998). Virtual Reality Training-Is It For You? Talk presented at the American Society for Training and Development International Conference, San Francisco, June 1998.
• 16: Curry I. Guinn, Mechanisms for mixed-initiative human-computer collaborative discourse, Proceedings of the 34th annual meeting on Association for Computational Linguistics, p.278-285, June 24-27, 1996, Santa Cruz, California [doi>10.3115/981863.981900]
• 17: Guinn, C.I., & Montoya, R.J. (1998). Natural Language Processing in Virtual Reality. Modern Simulation & Training, 6, 44--45.
• 18: Guye-Vuillième, A., Capin, T.K., Pandzic, I.S., Magnenat Thalmann, N., & Thalmann, D. (1999) Non-verbal Communication Interface for Collaborative Virtual Environments. The Virtual Reality Journal, Springer.
• 19: Hubal, R.C., & Frank, G.A. (2001). Interactive Training Applications using Responsive Virtual Human Technology. Proceedings of I/ITSEC, Orlando, FL, November 2001.
• 20: Hubal, R.C., Frank, G.A., & Guinn, C.I. (2000). AVATALK Virtual Humans for Training with Computer Generated Forces. Proceedings of CGF-BR. Institute for Simulation & Training: Orlando FL.
• 21: Hubal, R.C., Kizakevich, P.N., Guinn, C.I., Merino, K.D., & West, S.L. (2000). The Virtual Standardized Patient-Simulated Patient-Practitioner Dialogue for Patient Interview Training. In J.D. Westwood, H.M. Hoffman, G.T. Mogel, R. A. Robb, & D. Stredney (Eds.), Medicine Meets Virtual Reality. IOS Press and Ohmsha, Amsterdam, 133--138.
• 22: Johnson, W.L., Rickel, J.W., & Lester, J.C. (2000). Animated Pedagogical Agents: Face-to-Face Interaction in Interactive Learning Environments. International Journal of Artificial Intelligence in Education, 11, 47--78.
• 23: Kizakevich, P.N., McCartney, M.L., Nissman, D.B., Starko, K., & Smith, N.T. (1998). Virtual Medical Trainer: Patient Assessment and Trauma Care Simulator. Medicine Meets Virtual Reality-Art, Science, Technology: Healthcare (R)evolution, J.D. Westwood, H.M. Hoffman, D. Stredney, & S.J. Weghorst (Eds.), 309--315. IOS Press and Ohmsha: Amsterdam.
• 24: Klein, G. (1998). Sources of Power. Cambridge, MA: MIT Press.
• 25: Kline, C., & Blumberg, B. (1999). The Art and Science of Synthetic Character Design. Proceedings of the Symposium on AI and Creativity in Entertainment and Visual Art, Edinburgh.
• 26: Knapp, M.L., & Hall, J.A. (1997). Nonverbal Communication in Human Interaction (3rd ed.). Fort Worth: Harcourt Brace.
• 27: Krause, R., Steimer, E., Sänger-Alt, C., & Wagner, G. (1989). Facial Expression of Schizophrenic Patients and Their Interactions Partners. Psychiatry, 52, 1--11.
• 28: James C. Lester , Brian A. Stone , Gary D. Stelling, Lifelike Pedagogical Agents for Mixed-initiative Problem Solving in Constructivist Learning Environments, User Modeling and User-Adapted Interaction, v.9 n.1-2, p.1-44, April 1999 [doi>10.1023/A:1008374607830]
• 29: A. Bryan Loyall , Joseph Bates, Personality-rich believable agents that use language, Proceedings of the first international conference on Autonomous agents, p.106-113, February 05-08, 1997, Marina del Rey, California, United States [doi>10.1145/267658.267681]
• 30: Lundeberg, M., & Beskow, J. (1999). Developing a 3D-Agent for the August Dialogue System. AVSP Proceedings, Santa Cruz CA.
• 31: Moore, G. (2001). Talking Heads: Facial Animation in The Getaway. Gamasutra.
• 32: Olsen, D.E. (2001). The Simulation of a Human for Interpersonal Skill Training. In Proceedings of the 2001 Office of National Drug Control Policy (ONDCP) International Technology Symposium, June 25-28, 2001, San Diego CA.
• 33: Ortony, A., Clore, G. L., & Collins, A. (1988). The Cognitive Structure of Emotions. Cambridge: Cambridge University Press.
• 34: Plutchik, R. (1997). The Circumplex as a General Model of the Structure of Emotions and Personality. In R. Plutchik & H.R. Conte (Eds.) Circumplex Models of Personality and Emotions (pp. 17-45). American Psychological Association.
• 35: Shannon Pollard , Alan W. Biermann, A measure of semantic complexity for natural language systems, NAACL-ANLP 2000 Workshop on Syntactic and semantic complexity in natural language processing systems, p.42-46, April 30-30, 2000, Seattle, Washington [doi>10.3115/1117543.1117550]
• 36: Helmut Prendinger , Mitsuru Ishizuka, Social role awareness in animated agents, Proceedings of the fifth international conference on Autonomous agents, p.270-277, May 2001, Montreal, Quebec, Canada [doi>10.1145/375735.376307]
• 37: Ross, K.G., Pierce, L.G., Haltermann, J.A., & Ross, W.A. (1998). Preparing for the Instructional Technology Gap-A Constructivist Approach. Proceedings of I/ITSEC, Orlando FL.
• 38: Rousseau, D., & Hayes-Roth, B. (1997). Improvisational Synthetic Actors with Flexible Personalities. KSL Report #97-10, Stanford University.
• 39: Russell, J.A. (1997). How Shall an Emotion Be Called? In R. Plutchik & H.R. Conte (Eds.), Circumplex Models of Personality and Emotions (pp. 205--220). American Psychological Assn.
• 40: Shalif, I. (1991). The emotions and the dimensions of discrimination among them in daily life. Bar-Ilan University Ramat-Gan, Israel, unpublished dissertation.
• 41: Ronnie W. Smith , Steven A. Gordon, Effects of variable initiative on linguistic behavior in human-computer spoken natural language dialogue, Computational Linguistics, v.23 n.1, p.141-168, March 1997
• 42: Wiggins, J.S., (Ed.) (1996). The Five Factor Model of Personality: Theoretical Perspectives. New York: The Guilford Press.

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.1 Requirements/Specifications
          Subjects: Elicitation methods (e.g., rapid prototyping, interviews, JAD)

Additional Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.1 Requirements/Specifications
          Subjects: Methodologies (e.g., object-oriented, structured)

  E. Data
  E.1 DATA STRUCTURES
      Subjects: Graphs and networks

  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Natural language

  J. Computer Applications
  J.4 SOCIAL AND BEHAVIORAL SCIENCES
      Subjects: Psychology

General Terms:
Design, Documentation, Experimentation, Languages

Keywords:
agents, behavior modeling, interaction skills training, managing encounters with the mentally ill, responsive virtual humans