Sit straight (and tell me what I did today): a human posture alarm and activity summarization system

ABSTRACT

In this paper we present a novel system for monitoring a computer user's posture and activities in front of the computer (e.g., reading, speaking on the phone, etc.) for self-reporting. In our system, a camera and a microphone are placed in front of a computer work area (e.g., on top of the computer screen). The system monitors the computer user's postures and summarizes his or her activities. The system gives the user real time feedback on the goodness of his current posture, triggers alarms if the postures are not good postures, and generates summaries of postures and activities over a specified period of time (e.g., hours, days, months, etc.). All elements of the system are highly customizable: the user decides what "good" postures are, what alarms are triggered, if any, and what activity and posture summaries are generated. We present novel algorithms for posture measurement (using geometric features of the user's silhouette), and activity classification (using machine learning). Finally, we present experiments that show the feasibility of our approach, and discuss privacy issues and applications of the techniques presented (health monitoring, productivity analysis, and others).

REFERENCES

• 1: A Guide to Occupational Health and Safety in the New Zealand Sex Industry. Occupational Safety and Health Service, Department of Labor, Wellington, New Zealand, June 2004.
• 2: M.D. Abramoff, P.J. Magelhaes, and S.J. Ram, S.J. "Image Processing with ImageJ," Biophotonics International, volume 11, issue 7, pp. 36--42, 2004.
• 3: Piotr D. Adamczyk , Shamsi T. Iqbal , Brian P. Bailey, A method, system, and tools for intelligent interruption management, Proceedings of the 4th international workshop on Task models and diagrams, September 26-27, 2005, Gdansk, Poland [doi>10.1145/1122935.1122959]
• 4: A. Adamek, N.E. O'Connor, and G. Jones, "An Integrated Approach for Object Shape Registration and Modeling," in 2005 Intl. Workshop on Multimedia Information Retrieval in conjunction with SIGIR 2005, Rio de Janeiro, Brazil, 2005.
• 5: "Attentive User Interfaces," special issue, Communications of the ACM, (J. Vertegaal, ed.), Vol. 46, No. 3, March 2003.
• 6: O. Basir, D. Bullock, and E. Breza, "Visual classification and posture estimation of multiple vehicle occupants," US Patent #20040220705, Nov. 4, 2004.
• 7: B. Boulay, F. Bremond, and M. Thonnat, "Human Posture Recognition in Video Sequence," in pro. Joint IEEE International Workshop on VS-PETS, pp.23--29, Nice, France?Oct. 11-12?2003
• 8: M.-S. Chuang, and M.-S. Huang, "System and method for detecting unhealthy operation posture of using keyboard," US Patent #20030151595, Feb. 7, 2002.
• 9: M.D. Cordea, E.M. Petriu, N.D. Georganas, D.C.Petriu, and T.E. Whalen, "Real-time 2(1/2)-D Head Pose Recovery for Model-Based Video Coding," IEEE Transactions on Instrumentation and Measurement, Vol. 50, No. 4, August 2001.
• 10: M.P. Cooke, and G.J. Brown, "Interactive explorations in speech and hearing." J. Acoust. Soc. Japan (E), 20, 2, 89--97, 1999 (http://www.dcs.shef.ac.uk/~martin/).
• 11: R. Cucchiara, C. Grana, A. Prati, and R. Vezzani, "Probabilistic Posture Classification for Human-Behavior Analysis," IEEE Transactions on Systems, Man, and Cybernetics, Vol. 35, No.1, January 2005.
• 12: R. Donkin, "Typing Injury FAQ: Software Monitoring Tools," (http://www.faqs.org/faqs/typing-injury-faq/software), 1995, accessed Aug. 20. 2005.
• 13: A. Elgammal, V. Shet, Y. Yacoob, L. S. Davis, "Gesture Recognition using a Probabilistic Framework for Pose Matching," The Seventh International Conference on Control, Automation, Robotics and Vision, ICARCV 2002, Singapore in December 2-5, 2002.
• 14: Jim Gemmell , Roger Lueder , Gordon Bell, The MyLifeBits lifetime store, Proceedings of the 2003 ACM SIGMM workshop on Experiential telepresence, November 07, 2003, Berkeley, California [doi>10.1145/982484.982500]
• 15: S.R. Gunn and M.S. Nixon, "Snake Head Boundary Extraction using Global and Local Energy Minimisation," in proc. IEEE Int. Conf. on Pattern Recognition, pp. 581--585. Vienna, 1996.
• 16: Ismail Haritaoglu , David Harwood , Larry S. Davis, Ghost: A Human Body Part Labeling System Using Silhouettes, Proceedings of the 14th International Conference on Pattern Recognition-Volume 1, p.77, August 16-20, 1998
• 17: A. Hedge, and E.J. Ray, "Effects of an electronic height-adjustable worksurface on self-assessed musculoskeletal discomfort and productivity among computer workers," proceedings of the Human Factors and Ergonomics Society 48th Annual Meeting, New Orleans, Sept. 20-24, HFES, Santa Monica, 1091--1095, 2004.
• 18: A. Hedge, "Effects of Ergonomic Management Software on Employee Performance," Cornell Human Factors Laboratory Technical Report #/RP9991, 1991.
• 19: Hitoshi Ikeda , Masahiro Maeda , Noriji Kato , Hirotsugu Kashimura, Classification of human actions using face and hands detection, Proceedings of the 12th annual ACM international conference on Multimedia, October 10-16, 2004, New York, NY, USA [doi>10.1145/1027527.1027644]
• 20: Slobodan Ilic , Mathieu Salzmann , Pascal Fua, Implicit Surfaces Make for Better Silhouettes, Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Volume 1, p.1135-1141, June 20-26, 2005 [doi>10.1109/CVPR.2005.185]
• 21: Alejandro Jaimes , Shih-Fu Chang, Conceptual structures and computational methods for indexing and organization of visual information, 2003
• 22: A. Jaimes and N. Sebe, "Multimodal Human Computer Interaction: A Survey," IEEE International Workshop on Human-Computer Interaction (HCI 2005) in conjunction with IEEE International Conference on Computer Vision (ICCV 2005), Beijing, China, Oct. 15-21, 2005.
• 23: A. Jaimes, J. Liu, and N. Sebe, "Affective Meeting Video Analysis," in proc. IEEE ICME 2005, Amsterdam, The Netherlands, July 2005.
• 24: M. Kösch, A. C. B., and M. Turk, "Postural Comfort Zone for Reaching Gestures," in HFES Annual Meeting Notes, October 2003.
• 25: M.W. Lee and I. Cohen, "Human Upper Body Pose Estimation in Static Images, " in proc. ECCV 2004, Vol II, pp. 126-138, 2004.
• 26: J. Lee, B. Moghaddam, H. Pfister, and R. Machiraju, "Finding Optimal Views for 3D Face Shape Modeling," in proc. International Conference on Automatic Face and Gesture Recognition, pp. 31-36?Seoul, Korea, May 2004.
• 27: H. Nomura and T. Shima, "Image feature extraction apparatus, method of extracting image characteristic, monitoring and inspection system, exposure system, and interface system," US Patent #20020015526, Feb. 7, 2002.
• 28: R. Rosales and S. Sclaroff, "Inferring Body Pose Without Tracking Body Parts," IEEE Conference on Computer Vision and Pattern Recognition 2000, pp.721-727 vol.2?Hilton Head Island, USA, June 2000.
• 29: F. Sha and L. K. Saul, "Real-Time Pitch Determination of One or More Voices by Nonnegative Matrix Factorization," in L. K. Saul, Y. Weiss, and L. Bottou (eds.), Advances in Neural Information Processing Systems 17. MIT Press: Cambridge, MA, 2005.
• 30: Gregory Shakhnarovich , Paul Viola , Trevor Darrell, Fast Pose Estimation with Parameter-Sensitive Hashing, Proceedings of the Ninth IEEE International Conference on Computer Vision, p.750, October 13-16, 2003
• 31: C. Sminchisescu and A. Telea, "Human Pose Estimation From Silhouettes A Consistent Approach Using Distance Level Sets," WSCG International Conference on Computer Graphics, Visualization and Computer Vision, 2002.
• 32: Neville Stanton , Alan Hedge , Hal W. Hendrick , Eduardo Salas , Karel Brookhuis, Handbook of Human Factors and Ergonomics Methods, Taylor & Francis, Inc., Bristol, PA, 2004
• 33: T. Suzuki and D. Miwako, "Life support apparatus and method and method for providing advertisement information," US Patent #20010049471, Dec. 6, 2001.
• 34: Ian H. Witten , Eibe Frank, Data mining: practical machine learning tools and techniques with Java implementations, Morgan Kaufmann Publishers Inc., San Francisco, CA, 2000
• 35: Y. Zheng, "Method and apparatus for sensing body gesture, posture and movement," US Patent # 20040024312, Feb. 5, 2004.
• 36: Ergonomix (http://www.publicspace.net/ergonomix/)
• 37: Stretch Break (http://www.paratec.com/)
• 38: RSI guard (http://www.rsiguard.com/)
• 39: Ergotimer (http://www.tropsoft.com/ergotimer/ergonomics.htm)
• 40: Break Time (http://kadmi.com/products.html)
• 41: Protector of Health (http://www.olympsoft.com/)
• 42: Sonet Acoustic Privacy System (http://www.speechprivacysystems.com/)
• 43: http://www.ergoweb.com
• 44: http://www.hfes.org
• 45: http://www.m-w.com
• 46: http://ergonomicsinhealthcare.org

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.4 IMAGE PROCESSING AND COMPUTER VISION
      I.4.9 Applications

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Ergonomics

General Terms:
Algorithms, Human Factors, Measurement

Keywords:
computer vision, ergonomics, posture