Hand gesture recognition and virtual game control based on 3D accelerometer and EMG sensors

ABSTRACT

This paper describes a novel hand gesture recognition system that utilizes both multi-channel surface electromyogram (EMG) sensors and 3D accelerometer (ACC) to realize user-friendly interaction between human and computers. Signal segments of meaningful gestures are determined from the continuous EMG signal inputs. Multi-stream Hidden Markov Models consisting of EMG and ACC streams are utilized as decision fusion method to recognize hand gestures. This paper also presents a virtual Rubik's Cube game that is controlled by the hand gestures and is used for evaluating the performance of our hand gesture recognition system. For a set of 18 kinds of gestures, each trained with 10 repetitions, the average recognition accuracy was about 91.7% in real application. The proposed method facilitates intelligent and natural control based on gesture interaction.

REFERENCES

• 1: Asghari Oskoei M. and Hu H. Myoelectric control systems--A survey. Biomedical Signal Processing and Control, Vol. 2(4), 2007, pp. 275--294.
• 2: Helene Brashear , Thad Starner , Paul Lukowicz , Holger Junker, Using Multiple Sensors for Mobile Sign Language Recognition, Proceedings of the 7th IEEE International Symposium on Wearable Computers, p.45, October 21-23, 2003
• 3: Enrico Costanza , Samuel A. Inverso , Rebecca Allen, Toward subtle intimate interfaces for mobile devices using an EMG controller, Proceedings of the SIGCHI conference on Human factors in computing systems, April 02-07, 2005, Portland, Oregon, USA [doi>10.1145/1054972.1055039]
• 4: Jonghwa Kim , Stephan Mastnik , Elisabeth André, EMG-based hand gesture recognition for realtime biosignal interfacing, Proceedings of the 13th international conference on Intelligent user interfaces, January 13-16, 2008, Gran Canaria, Spain [doi>10.1145/1378773.1378778]
• 5: Mitra S. and Acharya T. Gesture Recognition: A Survey. IEEE Transactions on Systems, Man, and Cybernetics, Vol. 37(3), 2007, pp. 311--324.
• 6: Jani Mäntyjärvi , Juha Kela , Panu Korpipää , Sanna Kallio, Enabling fast and effortless customisation in accelerometer based gesture interaction, Proceedings of the 3rd international conference on Mobile and ubiquitous multimedia, p.25-31, October 27-29, 2004, College Park, Maryland [doi>10.1145/1052380.1052385]
• 7: Rubik's Cube http://en.wikipedia.org/wiki/Rubik's_cube
• 8: Sherrill D.M., Bonato P., De Luca C.J. A neural network approach to monitor motor activities. In Proceedings of the Second Joint EMBS/BMES Conference, vol.1, Houston, USA, 2002, pp. 52--53.
• 9: Tamura S., Iwano K., and Furui S. A stream-weight optimization method for audio-visual speech recognition using multi-stream HMMs. In Proc. IEEE Int. Conf. Acoustics, Speech, and Signal Processing (ICASSP '04), vol. 1, 2004, pp. I-857--60.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Input devices and strategies (e.g., mouse, touchscreen)

General Terms:
Algorithms, Measurement

Keywords:
accelerometer, electromyogram, gesture recognition, human computer interaction