Posture and activity silhouettes for self-reporting, interruption management, and attentive interfaces

ABSTRACT

In this paper we present a novel system for monitoring a computer user's posture and activities in front of the computer (e.g., reading, speaking on the phone, etc.) for self-reporting. In our system, a camera and a microphone are placed in front of a computer work area (e.g., on top of the computer screen). The system can be used as a component in an attentive interface, or for giving the user real time feedback on the goodness of his current posture, and generating summaries of postures and activities over a specified period of time (e.g., hours, days, months, etc.). All elements of the system are highly customizable: the user decides what "good" postures are, what alarms and interruptions are triggered, if any, and what activity and posture summaries are generated. We present novel algorithms for posture measurement (using geometric features of the user's silhouette), and activity classification (using machine learning). Finally, we present experiments that show the feasibility of our approach.

REFERENCES

• 1: A Guide to Occupational Health and Safety in the New Zealand Sex Industry. Occupational Safety and Health Serv., Dept. of Labor, Wellington, New Zealand, June 2004.
• 2: M.D. Abramoff, P.J. Magelhaes, and S.J. Ram, S.J. "Image Processing with ImageJ," Biophotonics International, volume 11, issue 7, pp. 36--42, 2004.
• 3: Piotr D. Adamczyk , Shamsi T. Iqbal , Brian P. Bailey, A method, system, and tools for intelligent interruption management, Proceedings of the 4th international workshop on Task models and diagrams, September 26-27, 2005, Gdansk, Poland [doi>10.1145/1122935.1122959]
• 4: A. Adamek, N.E. O'Connor, and G. Jones, "An Integrated Approach for Object Shape Registration and Modeling," in 2005 Intl. Workshop on Multimedia Info. Retrieval in conjunction with SIGIR 2005, Rio de Janeiro, Brazil, 2005.
• 5: O. Basir, D. Bullock, and E. Breza, "Visual classification and posture estimation of multiple vehicle occupants," US Patent #20040220705, Nov. 4, 2004.
• 6: B. Boulay, F. Bremond, and M. Thonnat, "Human Posture Recognition in Video Sequence," in pro. Joint IEEE International Workshop on VS-PETS, pp.23-29, Nice, France?Oct. 11-12?2003.
• 7: Daniel Chen , Roel Vertegaal, Using mental load for managing interruptions in physiologically attentive user interfaces, CHI '04 extended abstracts on Human factors in computing systems, April 24-29, 2004, Vienna, Austria [doi>10.1145/985921.986103]
• 8: M.-S. Chuang, and M.-S. Huang, "System and method for detecting unhealthy operation posture of using keyboard," US Patent #20030151595, Feb. 7, 2002.
• 9: M.D. Cordea, E.M. Petriu, N.D. Georganas, D.C.Petriu, and T.E. Whalen, "Real-time 2(1/2)-D Head Pose Recovery for Model-Based Video Coding," IEEE Trans. on Instrument. and Measurement, Vol. 50, No. 4, August 2001.
• 10: R. Cucchiara, C. Grana, A. Prati, and R. Vezzani, "Probabilistic Posture Classification for Human-Behavior Analysis," IEEE Transactions on Systems, Man, and Cybernetics, Vol. 35, No.1, January 2005.
• 11: R. Donkin, "Typing Injury FAQ: Software Monitoring Tools," (http://www.faqs.org/faqs/typing-injury-faq/software), 1995, accessed Aug. 20. 2005.
• 12: A. Elgammal, V. Shet, Y. Yacoob, L. S. Davis, "Gesture Recognition using a Probabilistic Framework for Pose Matching," The 7th Int. Conf. on Control, Automation, Robotics and Vision, ICARCV, Singapore in Dec. 2-5, 2002.
• 13: Jim Gemmell , Roger Lueder , Gordon Bell, The MyLifeBits lifetime store, Proceedings of the 2003 ACM SIGMM workshop on Experiential telepresence, November 07, 2003, Berkeley, California [doi>10.1145/982484.982500]
• 14: S.R. Gunn and M.S. Nixon, "Snake Head Boundary Extraction using Global and Local Energy Minimisation," in proc. IEEE Int. Conf. on Pattern Recognition, pp. 581--585. Vienna, 1996.
• 15: Ismail Haritaoglu , David Harwood , Larry S. Davis, Ghost: A Human Body Part Labeling System Using Silhouettes, Proceedings of the 14th International Conference on Pattern Recognition-Volume 1, p.77, August 16-20, 1998
• 16: A. Hedge, and E.J. Ray, "Effects of an electronic height-adjustable worksurface on self-assessed musculoskeletal discomfort and productivity among computer workers," proceedings of the Human Factors and Ergonomics Society 48th Annual Meeting, New Orleans, Sept. 20-24, HFES, Santa Monica, 1091--1095, 2004.
• 17: A. Hedge, "Effects of Ergonomic Management Software on Employee Performance," Cornell Human Factors Laboratory Technical Report #/RP9991, 1991.
• 18: Eric Horvitz , Paul Koch , Johnson Apacible, BusyBody: creating and fielding personalized models of the cost of interruption, Proceedings of the 2004 ACM conference on Computer supported cooperative work, November 06-10, 2004, Chicago, Illinois, USA [doi>10.1145/1031607.1031690]
• 19: Hitoshi Ikeda , Masahiro Maeda , Noriji Kato , Hirotsugu Kashimura, Classification of human actions using face and hands detection, Proceedings of the 12th annual ACM international conference on Multimedia, October 10-16, 2004, New York, NY, USA [doi>10.1145/1027527.1027644]
• 20: Slobodan Ilic , Mathieu Salzmann , Pascal Fua, Implicit Surfaces Make for Better Silhouettes, Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Volume 1, p.1135-1141, June 20-26, 2005 [doi>10.1109/CVPR.2005.185]
• 21: A. Jaimes and N. Sebe, "Multimodal Human Computer Interaction: A Survey," IEEE Int. Workshop on Human-Computer Interaction at IEEE Int. Conf. on Computer Vision (ICCV 2005), Beijing, China, Oct. 15-21, 2005.
• 22: A. Jaimes, J. Liu, and N. Sebe, "Affective Meeting Video Analysis," in proc. IEEE ICME 2005, Amsterdam, The Netherlands, July 2005.
• 23: Alejandro Jaimes, Sit straight (and tell me what I did today): a human posture alarm and activity summarization system, Proceedings of the 2nd ACM workshop on Continuous archival and retrieval of personal experiences, November 11-11, 2005, Hilton, Singapore [doi>10.1145/1099083.1099087]
• 24: M. Kölsch, A. C. B., and M. Turk, "Postural Comfort Zone for Reaching Gestures," in HFES Annual Meeting Notes, October 2003.
• 25: M.W. Lee and I. Cohen, "Human Upper Body Pose Estimation in Static Images, " in proc. ECCV 2004, Vol II, pp. 126--138, 2004.
• 26: J. Lee, B. Moghaddam, H. Pfister, and R. Machiraju, "Finding Optimal Views for 3D Face Shape Modeling," in proc. International Conference on Automatic Face and Gesture Recognition, pp. 31--36?Seoul, Korea, May 2004.
• 27: H. Nomura and T. Shima, "Image feature extraction apparatus, method of extracting image characteristic, monitoring and inspection system, exposure system, and interface system," US Patent #20020015526, Feb. 7, 2002.
• 28: R. Rosales and S. Sclaroff, "Inferring Body Pose Without Tracking Body Parts," in IEEE Conf. on Comp. Vision and Pattern Recognition, Hilton Head Island, USA, June 2000.
• 29: Gregory Shakhnarovich , Paul Viola , Trevor Darrell, Fast Pose Estimation with Parameter-Sensitive Hashing, Proceedings of the Ninth IEEE International Conference on Computer Vision, p.750, October 13-16, 2003
• 30: C. Sminchisescu and A. Telea, "Human Pose Estimation From Silhouettes A Consistent Approach Using Distance Level Sets," WSCG International Conference on Computer Graphics, Visualization and Computer Vision, 2002.
• 31: Paul Smith , Niels da Vitoria Lobo , Mubarak Shah, TemporalBoost for Event Recognition, Proceedings of the Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1, p.733-740, October 17-20, 2005 [doi>10.1109/ICCV.2005.234]
• 32: Neville Stanton , Alan Hedge , Hal W. Hendrick , Eduardo Salas , Karel Brookhuis, Handbook of Human Factors and Ergonomics Methods, Taylor & Francis, Inc., Bristol, PA, 2004
• 33: T. Suzuki and D. Miwako, "Life support apparatus and method and method for providing advertisement information," US Patent #20010049471, Dec. 6, 2001.
• 34: R. Vertegaal (ed.), "Attentive User Interfaces" special issue, Communications of the ACM, Vol. 46, No. 3, March, 2003.
• 35: Ian H. Witten , Eibe Frank, Data mining: practical machine learning tools and techniques with Java implementations, Morgan Kaufmann Publishers Inc., San Francisco, CA, 2000
• 36: Y. Zheng, "Method and apparatus for sensing body gesture, posture and movement," US Patent # 20040024312, Feb. 5, 2004.
• 37: Ergonomix (http://www.publicspace.net/ergonomix/)
• 38: Stretch Break (http://www.paratec.com/)
• 39: RSI guard (http://www.rsiguard.com/)
• 40: Ergotimer (http://www.tropsoft.com/ergotimer/ergonomics.htm)
• 41: Break Time (http://kadmi.com/products.html)
• 42: Protector of Health (http://www.olympsoft.com/)

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.4 IMAGE PROCESSING AND COMPUTER VISION
      I.4.9 Applications

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Ergonomics

General Terms:
Algorithms, Human Factors, Measurement

Keywords:
computer vision, ergonomics, posture