Human computing and machine understanding of human behavior

ABSTRACT

A widely accepted prediction is that computing will move to the background, weaving itself into the fabric of our everyday living spaces and projecting the human user into the foreground. If this prediction is to come true, then next generation computing, which we will call human computing, should be about anticipatory user interfaces that should be human-centered, built for humans based on human models. They should transcend the traditional keyboard and mouse to include natural, human-like interactive functions including understanding and emulating certain human behaviors such as affective and social signaling. This article discusses a number of components of human behavior, how they might be integrated into computers, and how far we are from realizing the front end of human computing, that is, how far are we from enabling computers to understand human behavior.

REFERENCES

Note: OCR errors may be found in this Reference List extracted from the full text article. ACM has opted to expose the complete List rather than only correct and linked references.

	1	Emile Aarts, Ambient intelligence drives open innovation, interactions, v.12 n.4, July + August 2005 [doi>10.1145/1070960.1070996]
	2	Ambady, N. and Rosenthal, R. Thin slices of expressive behavior as predictors of interpersonal consequences: A meta-analysis. Psychological Bulletin, 111, 2 (Feb. 1992), 256--274.
	3	Sileye O. Ba , Jean-Marc Odobez, A Probabilistic Framework for Joint Head Tracking and Pose Estimation, Proceedings of the Pattern Recognition, 17th International Conference on (ICPR'04) Volume 4, p.264-267, August 23-26, 2004 [doi>10.1109/ICPR.2004.87]
	4	Marian Stewart Bartlett , Gwen Littlewort , Mark Frank , Claudia Lainscsek1 , Ian Fasel , Javier Movellan, Fully Automatic Facial Action Recognition in Spontaneous Behavior, Proceedings of the 7th International Conference on Automatic Face and Gesture Recognition (FGR06), p.223-230, April 10-12, 2006 [doi>10.1109/FGR.2006.55]
	5	Manuele Bicego , Marco Cristani , Vittorio Murino, Unsupervised scene analysis: a hidden Markov model approach, Computer Vision and Image Understanding, v.102 n.1, p.22-41, April 2006 [doi>10.1016/j.cviu.2005.09.011]
	6	Bobick, A. F. Movement, activity and action: The role of knowledge in the perception of motion. Philosophical Trans. Roy. Soc. London B, 352, 1358 (Aug. 1997), 1257--1265.
	7	Kevin W. Bowyer , Kyong Chang , Patrick Flynn, A survey of approaches and challenges in 3D and multi-modal 3D + 2D face recognition, Computer Vision and Image Understanding, v.101 n.1, p.1-15, January 2006 [doi>10.1016/j.cviu.2005.05.005]
	8	Buxton, H. Learning and understanding dynamic scene activity: a review. Image & Vision Computing, 21, 1 (Jan. 2003), 125--136.
	9	Cacioppo, J. T., Berntson, G. G., Larsen, J. T., Poehlmann, K. M. and Ito, T. A. The psychophysiology of emotion. In Handbook of Emotions. Lewis, M. and Haviland-Jones, J. M., Eds. Guilford Press, New York, 2000, 173--191.
	10	Cheng-Chin Chiang , Chi-Jang Huang, A robust method for detecting arbitrarily tilted human faces in color images, Pattern Recognition Letters, v.26 n.16, p.2518-2536, December 2005 [doi>10.1016/j.patrec.2005.05.008]
	11	Costa, M., Dinsbach, W., Manstead, A. S. R. and Bitti, P. E. R. Social presence, embarrassment, and nonverbal behavior. Journal of Nonverbal Behavior, 25, 4 (Dec. 2001), 225--240.
	12	Coulson, M. Attributing emotion to static body postures: Recognition accuracy, confusions, & viewpoint dependence. J. Nonverbal Behavior, 28, 2 (Jun. 2004), 117--139.
	13	Li Deng , Xuedong Huang, Challenges in adopting speech recognition, Communications of the ACM, v.47 n.1, January 2004 [doi>10.1145/962081.962108]
	14	Dey, A. K., Abowd, G. D. and Salber, D. A conceptual framework and a toolkit for supporting the rapid prototyping of context-aware applications. J. Human-Computer Interaction, 16, 2/4 (Dec. 2001), 97--166.
	15	Duchowski, A. T. A breadth-first survey of eye-tracking applications. Behavior Research Methods, Instruments and Computing, 34, 4 (Nov. 2002), 455--470.
	16	Ekman, P. and Friesen, W. F. The repertoire of nonverbal behavioral categories -- origins, usage, and coding. Semiotica, 1, 1969, 49--98.
	17	Ekman, P., Friesen, W.V. and Hager, J. C. Facial Action Coding System. A Human Face, Salt Lake City, 2002.
	18	Rana El Kaliouby , Peter Robinson, Real-Time Inference of Complex Mental States from Facial Expressions and Head Gestures, Proceedings of the 2004 Conference on Computer Vision and Pattern Recognition Workshop (CVPRW'04) Volume 10, p.154, June 27-July 02, 2004
	19	Fridlund, A. J. The new ethology of human facial expression. The psychology of facial expression. Russell, J. A. and Fernandez-Dols, J. M., Eds. Cambridge University Press, Cambridge, UK, 1997, 103--129.
	20	G. W. Furnas , T. K. Landauer , L. M. Gomez , S. T. Dumais, The vocabulary problem in human-system communication, Communications of the ACM, v.30 n.11, p.964-971, Nov. 1987 [doi>10.1145/32206.32212]
	21	Gatica-Perez, D., McCowan, I., Zhang, D. and Bengio, S. Detecting group interest level in meetings. In Proc. Int'l Conf. Acoustics, Speech & Signal Processing, vol. 1, 489--492, 2005.
	22	Gibson, K. R. and Ingold, T., Eds. Tools, Language and Cognition in Human Evolution. Cambridge University Press, Cambridge, UK, 1993.
	23	Gu, H. and Ji, Q. Information extraction from image sequences of real-world facial expressions. Machine Vision and Applications, 16, 2 (Feb. 2005), 105--115.
	24	Gunes, H. and Piccardi, M. Affect Recognition from Face and Body: Early Fusion vs. Late Fusion, In Proc. Int'l Conf. Systems, Man and Cybernetics, 3437--3443, 2005.
	25	Haykin, S. and de Freitas, N., Eds. Special Issue on Sequential State Estimation. Proceedings of the IEEE, 92, 3 (Mar. 2004), 399--574.
	26	Robust Real-Time Detection, Tracking, and Pose Estimation of Faces in Video Streams, Proceedings of the Pattern Recognition, 17th International Conference on (ICPR'04) Volume 3, p.965-968, August 23-26, 2004 [doi>10.1109/ICPR.2004.797]
	27	Izard, C. E. Emotions and facial expressions: A perspective from Differential Emotions Theory. In The psychology of facial expression. Russell, J. A. and Fernandez-Dols, J. M., Eds. Cambridge University Press, Cambridge, UK, 1997, 103--129.
	28	Anil K. Jain , Arun Ross, Multibiometric systems, Communications of the ACM, v.47 n.1, January 2004 [doi>10.1145/962081.962102]
	29	Juslin, P. N. and Scherer, K. R. Vocal expression of affect. In The New Handbook of Methods in Nonverbal Behavior Research. Harrigan, J., Rosenthal, R. and Scherer, K., Eds. Oxford University Press, Oxford, UK, 2005.
	30	Keltner, D. and Ekman, P. Facial expression of emotion. In Handbook of Emotions, Lewis, M., and Haviland-Jones, J. M. Eds. The Guilford Press, New York, 2000, pp. 236--249.
	31	Anil K. Jain , Stan Z. Li, Handbook of Face Recognition, Springer-Verlag New York, Inc., Secaucus, NJ, 2005
	32	Lisetti, C. L. and Schiano, D. J. Automatic facial expression interpretation: Where human-computer interaction, AI and cognitive science intersect. Pragmatics and Cognition, 8, 1 (Jan. 2000), 185--235.
	33	Ludo Maat , Maja Pantic, Gaze-X: adaptive affective multimodal interface for single-user office scenarios, Proceedings of the 8th international conference on Multimodal interfaces, November 02-04, 2006, Banff, Alberta, Canada [doi>10.1145/1180995.1181032]
	34	Matos, S., Birring, S. S., Pavord, I. D. and Evans, D. H. Detection of cough signals in continuous audio recordings using HMM. IEEE Trans. Biomedical Engineering, 53, 6 (June 2006), 1078--1083.
	35	Anton Nijholt , Thomas Rist , Kees Tuijnenbreijer, Lost in ambient intelligence?, CHI '04 extended abstracts on Human factors in computing systems, April 24-29, 2004, Vienna, Austria [doi>10.1145/985921.986202]
	36	Nijholt, A., de Ruyter, B., Heylen, D. and Privender, S. Social Interfaces for Ambient Intelligence Environments. Chapter 14 in: True Visions: The Emergence of Ambient Intelligence. Aarts, E. and Encarnaçao, J., Eds. Springer, New York, 2006, 275--289.
	37	Anton Nijholt , David Traum, The virtuality continuum revisited, CHI '05 extended abstracts on Human factors in computing systems, April 02-07, 2005, Portland, OR, USA [doi>10.1145/1056808.1057123]
	38	Harriet J. Nock , Giridharan Iyengar , Chalapathy Neti, Multimodal processing by finding common cause, Communications of the ACM, v.47 n.1, January 2004 [doi>10.1145/962081.962105]
	39	Donald A. Norman, Human-centered design considered harmful, interactions, v.12 n.4, July + August 2005 [doi>10.1145/1070960.1070976]
	40	Oudeyer Pierre-Yves, The production and recognition of emotions in speech: features and algorithms, International Journal of Human-Computer Studies, v.59 n.1-2, p.157-183, July 2003 [doi>10.1016/S1071-5819(02)00141-6]
	41	Oviatt, S. User-centered modeling and evaluation of multimodal interfaces. Proceedings of the IEEE, 91, 9 (Sep. 2003), 1457--1468.
	42	Pal, P., Iyer, A. N. and Yantorno, R. E. Emotion detection from infant facial expressions and cries. In Proc. Int'l Conf. Acoustics, Speech & Signal Processing, 2, 721--724, 2006.
	43	Pantic, M. and Patras, I. Dynamics of Facial Expressions -- Recognition of Facial Actions and their Temporal Segments from Face Profile Image Sequences. IEEE Trans. Systems, Man, and Cybernetics, Part B, 36, 2 (Apr. 2006), 433--449.
	44	Pantic, M. and Rothkrantz, L. J. M. Toward an Affect-Sensitive Multimodal Human-Computer Interaction. Proceedings of the IEEE, 91, 9 (Sep. 2003), 1370--1390.
	45	Pantic, M., Valstar, M. F., Rademaker, R. and Maat, L. Web-based database for facial expression analysis. In Proc. Int'l Conf. Multimedia and Expo, 2005. (www.mmifacedb.com)
	46	Alex (Sandy) Pentland, Socially Aware Computation and Communication, Computer, v.38 n.3, p.33-40, March 2005 [doi>10.1109/MC.2005.104]
	47	Russell, J. A. and Fernandez-Dols, J. M., Eds. The psychology of facial expression. Cambridge University Press, Cambridge, UK, 1997.
	48	Russell, J. A., Bachorowski, J. A. and Fernandez-Dols, J. M. Facial and Vocal Expressions of Emotion. Annual Review of Psychology, 54, (2003), 329--349.
	49	Z. M. Ruttkay , D. Reidsma , A. Nijholt, Human computing, virtual humans and artificial imperfection, Proceedings of the 8th international conference on Multimodal interfaces, November 02-04, 2006, Banff, Alberta, Canada [doi>10.1145/1180995.1181033]
	50	Peter Sand , Seth Teller, Particle Video: Long-Range Motion Estimation using Point Trajectories, Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, p.2195-2202, June 17-22, 2006 [doi>10.1109/CVPR.2006.219]
	51	Scanlon, P. and Reilly, R. B. Feature analysis for automatic speech reading. In Proc. Int'l Workshop Multimedia Signal Processing, 2001, 625--630.
	52	Sharma, R., Yeasin, M., Krahnstoever, N., Rauschert, I., Cai, G., Maceachren, A. M. and Sengupta, K. Speech-gesture driven multimodal interfaces for crisis management. Proceedings of the IEEE, 91, 9 (Sep. 2003), 1327--1354.
	53	Song, M., Bu, J., Chen, C. and Li, N. Audio-visual based emotion recognition -- A new approach. In Proc. Int'l Conf. Computer Vision and Pattern Recognition, 2004, 1020--1025.
	54	Thad Starner, The Challenges of Wearable Computing: Part 1, IEEE Micro, v.21 n.4, p.44-52, July 2001 [doi>10.1109/40.946681]
	55	Stein, B. and Meredith, M. A. The Merging of Senses. MIT Press, Cambridge, USA, 1993.
	56	Bjorn Stenger , Arasanathan Thayananthan , Philip H. S. Torr , Roberto Cipolla, Model-Based Hand Tracking Using a Hierarchical Bayesian Filter, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.28 n.9, p.1372-1384, September 2006 [doi>10.1109/TPAMI.2006.189]
	57	Daniel M. Russell , Norbert A. Streitz , Terry Winograd, Building disappearing computers, Communications of the ACM, v.48 n.3, March 2005 [doi>10.1145/1047671.1047702]
	58	Traum, D. Back end of human computing: Multimodal and multi-party interaction management. In Proc. Int'l Conf. Multimodal Interfaces, 2006.
	59	Truong, K. P. and van Leeuwen, D. A. Automatic detection of laughter. In Proc. Interspeech Euro. Conf., 485--488, 2005.
	60	Valstar, M. F. and Pantic, M. Biologically vs. logic inspired encoding of facial actions and emotions in video. In Proc. Int'l Conf. on Multimedia and Expo, 2006.
	61	Michel Valstar , Maja Pantic, Fully Automatic Facial Action Unit Detection and Temporal Analysis, Proceedings of the 2006 Conference on Computer Vision and Pattern Recognition Workshop, p.149, June 17-22, 2006 [doi>10.1109/CVPRW.2006.85]
	62	Michel F. Valstar , Maja Pantic , Zara Ambadar , Jeffrey F. Cohn, Spontaneous vs. posed facial behavior: automatic analysis of brow actions, Proceedings of the 8th international conference on Multimodal interfaces, November 02-04, 2006, Banff, Alberta, Canada [doi>10.1145/1180995.1181031]
	63	Paul Viola , Michael J. Jones, Robust Real-Time Face Detection, International Journal of Computer Vision, v.57 n.2, p.137-154, May 2004 [doi>10.1023/B:VISI.0000013087.49260.fb]
	64	Wang, J. J. and Singh, S. Video analysis of human dynamics -- a survey. Real Time Imaging, 9, 5 (Oct. 2003), 321--346.
	65	Wang, L., Hu, W. and Tan, T. Recent developments in human motion analysis. Pattern Recognition 36, 3 (Mar. 2003), 585--601.
	66	Weiser, M. The Computer for the Twenty-First Century. Scientific American, 265, 3 (Sep. 1991), 94--104.
	67	Ming-Hsuan Yang , David J. Kriegman , Narendra Ahuja, Detecting Faces in Images: A Survey, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.24 n.1, p.34-58, January 2002 [doi>10.1109/34.982883]
	68	Shumin Zhai , Victoria Bellotti, Introduction to sensing-based interaction, ACM Transactions on Computer-Human Interaction (TOCHI), v.12 n.1, p.1-2, March 2005 [doi>10.1145/1057237.1057238]
	69	W. Zhao , R. Chellappa , P. J. Phillips , A. Rosenfeld, Face recognition: A literature survey, ACM Computing Surveys (CSUR), v.35 n.4, p.399-458, December 2003 [doi>10.1145/954339.954342]
	70	Zhihong Zeng , Yuxiao Hu , Yun Fu , Thomas S. Huang , Glenn I. Roisman , Zhen Wen, Audio-visual emotion recognition in adult attachment interview, Proceedings of the 8th international conference on Multimodal interfaces, November 02-04, 2006, Banff, Alberta, Canada [doi>10.1145/1180995.1181028]
	71	BTT Survey on Alternative Biometrics. Biometric Technology Today, 14, 3 (Mar. 2006), 9--11.
	72	Humaine Portal: http://emotion-research.net/wiki/Databases
	73	MMUA: http://mmua.cs.ucsb.edu/
	74	Praat: http://www.praat.org