Multi modal gesture identification for HCI using surface EMG

ABSTRACT

Gesture and Speech comprise the most important modalities of human interaction. There has been a considerable amount of research attempts at incorporating these modalities for natural HCI. This involves challenge ranging from the low level signal processing of multi-modal input to the high level interpretation of natural speech and gesture in HCI. This paper proposes novel methods to recognize the hand gestures and unvoiced utterances using surface Electromyogram (sEMG) signals originating from different muscles. The focus of this work is to establish a simple, yet robust system that can be integrated to identify subtle complex hand gestures and unvoiced speech commands for control of prosthesis and other computer assisted devices. The proposed multi-modal system is able to identify the hand gestures and silent utterances using Independent Component Analysis (ICA) and Integral RMS (IRMS) of sEMG respectively. Training of the sEMG features was done using a designed ANN architecture and the results reported with overall recognition accuracy of 90.33%.

REFERENCES

• 1: Chan, D. C., Englehart, K., Hudgins, B., and Lovely, D. F. 2002. A multi-expert speech recognition system using acoustic and myoelectric signals. In Proceedings of the 24th Annual Conference and the Annual Fall Meeting of the Biomedical Engineering Society EMBS/BMES, Canada, 72--73.
• 2: Chan, A. D. C., and Englehart, K. B. 2005. Continuous myoelectric control for powered prostheses using hidden markov models. IEEE Trans. Biomed. Engg. 52, 1 (2005), 121--124.
• 3: Fistre, J., and Tanaka, A. 2002. Real time EMG gesture recognition for consumer electronics device control. Presented at Sony CSL Paris Open House, Paris, France. http://www.csl.sony.fr/atau/gesture/.
• 4: Fridlund, A. J., and Cacioppo, J. T. 1986. Guidelines for Human Electromyographic research. J. Psych. 23, 4 (1986), 567--589.
• 5: Hyvarinen, A., Karhunen, J., and Oja, E. 2001. Independent Component Analysis. John Wiley, New York.
• 6: Kermani, M. Z., Wheeler, B. C., Badie, K., and Hashemi, R. M. 1995. EMG Feature Evaluation for Movement Control of Upper Extremity Prostheses, IEEE Trans. Rehab. Engg. 2 (1995), 1267--1271.
• 7: Kumar, S., Kumar, D. K., Alemu, M., and Burry, M. 2004. EMG based voice recognition. In Proceedings of Sensor Networks and Information Processing. Melbourne, Australia.
• 8: Lapatki, G., Stegeman, D. F. and Jonas, I. E. 2003. A surface EMG electrode for the simultaneous observation of multiple facial muscles. J. Neuro. 123, 2 (2003), 117--128.
• 9: Hiroyuki Manabe , Akira Hiraiwa , Toshiaki Sugimura, "Unvoiced speech recognition using EMG - mime speech recognition", CHI '03 extended abstracts on Human factors in computing systems, April 05-10, 2003, Ft. Lauderdale, Florida, USA [doi>10.1145/765891.765996]
• 10: Parsons, T. W. 1986. Voice and speech processing.
• 11: Vladimir I. Pavlovic , Rajeev Sharma , Thomas S. Huang, Visual Interpretation of Hand Gestures for Human-Computer Interaction: A Review, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.19 n.7, p.677-695, July 1997 [doi>10.1109/34.598226]
• 12: Rehg, J. M. and Kanade, T. 1996. Vision-based hand tracking for human-computer interaction. In Proceedings of the IEEE Workshop on Motion of Non-Rigid and Articulated Objects, 16--22.
• 13: Schlenzig, J., Hunter, E., and Jain, R. 1994. Vision based hand gesture interpretation using recursive estimation. In Proceedings of Twenty-eighth Asilomar Conference on Signals, Systems and Computers, 1267--1271.
• 14: Trejo, L. J., Wheeler, K. R., Jorgensen, C. C., Rosipal, R., Clanton, T. S., Matthews, B., Hibbs, A. D., Matthews, R., and Krupka, M. 2003. Multimodal neuroelectric interface development. IEEE Trans. Neural Sys. Rehab. Engg. 11, 2 (2003), 199--204.
• 15: Kevin R. Wheeler , Charles C. Jorgensen, Gestures as Input: Neuroelectric Joysticks and Keyboards, IEEE Pervasive Computing, v.2 n.2, p.56-61, April 2003 [doi>10.1109/MPRV.2003.1203754]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.1 MODELS AND PRINCIPLES
      H.1.2 User/Machine Systems

General Terms:
Human Factors, Performance, Reliability

Keywords:
independent component analysis, multi-modal HCI, surface electromyography