Multimodal archiving, real-time annotation and information visualization in a biofeedback system for stroke patient rehabilitation

ABSTRACT

In this paper we present our work on a system to support real-time multimodal archiving, collaborative annotation and offline information visualization for a biofeedback stroke-rehabilitation application. Our archiving / annotation / visualization system can play a critical role in the long-term biofeedback stroke therapy by supporting cooperative data analysis and media feedback as well as by providing the therapist with insight into computing-supported therapy. There are three contributions of this paper: (a) the design of a robust archiving system that archives in real time parametric model data (motion capture, motion analysis and audio / visual synthesis parameters) as well as audio / video from the biofeedback environment. (b) a web-based annotation tool designed with low cognitive load (c) a hierarchical information visualization tool that enables the therapist and other team members to examine quantitative motion analysis of subject performance with the context of media feedback, thus enabling collaborative insights. Our user studies indicate that the system performs well.

REFERENCES

• 1: Max/MSP (media tool). http://www.cycling74.com.
• 2: J. J. Cadiz , Anop Gupta , Jonathan Grudin, Using Web annotations for asynchronous collaboration around documents, Proceedings of the 2000 ACM conference on Computer supported cooperative work, p.309-318, December 2000, Philadelphia, Pennsylvania, United States [doi>10.1145/358916.359002]
• 3: Y. Chen, H. Huang, W. Xu, R. Wallis, H. Sundaram, et al. (2006). The Design Of A Real-Time, Multimodal Biofeedback System For Stroke Patient Rehabilitation, Proc. SIG ACM Multimedia 2006, also AME-TR-2006-07, Oct. 2006, Santa Barbara, CA.
• 4: Sean E. Ellis , Dennis P. Groth, A collaborative annotation system for data visualization, Proceedings of the working conference on Advanced visual interfaces, May 25-28, 2004, Gallipoli, Italy [doi>10.1145/989863.989938]
• 5: Jim Gemmell , Gordon Bell , Roger Lueder , Steven Drucker , Curtis Wong, MyLifeBits: fulfilling the Memex vision, Proceedings of the tenth ACM international conference on Multimedia, December 01-06, 2002, Juan-les-Pins, France [doi>10.1145/641007.641053]
• 6: S. Hodges, L. Williams, E. Berry, S. Izadi, J. Srinivasan, et al. SenseCam: a Retrospective Memory Aid, Proc. 8th International Conference on Ubicomp 2006, Sep. 2006, Orange County, CA.
• 7: M. Holden and T. Dyar (2002). Virtual environment training: a new tool for neurorehabilitation. Neurology Report 26(2): 62--72.
• 8: D. Jack, R. Boian, A. S. Merians, M. Tremaine, G. C. Burdea, et al. (2001). Virtual reality-enhanced stroke rehabilitation. IEEE Trans Neural Syst Rehabil Eng 9(3): 308--318.
• 9: Robert E. Kraut , Darren Gergle , Susan R. Fussell, The use of visual information in shared visual spaces: informing the development of virtual co-presence, Proceedings of the 2002 ACM conference on Computer supported cooperative work, November 16-20, 2002, New Orleans, Louisiana, USA [doi>10.1145/587078.587084]
• 10: Jiazhi Ou , Lui Min Oh , Jie Yang , Susan R. Fussell, Effects of task properties, partner actions, and message content on eye gaze patterns in a collaborative task, Proceedings of the SIGCHI conference on Human factors in computing systems, April 02-07, 2005, Portland, Oregon, USA [doi>10.1145/1054972.1055005]
• 11: Catherine Plaisant , Brett Milash , Anne Rose , Seth Widoff , Ben Shneiderman, LifeLines: visualizing personal histories, Proceedings of the SIGCHI conference on Human factors in computing systems: common ground, p.221-ff., April 13-18, 1996, Vancouver, British Columbia, Canada [doi>10.1145/238386.238493]
• 12: D. Quan, D. Huynh and D. R. Karger (2003). Haystack: A Platform for Authoring End User Semantic Web Applications, 2nd International Semantic Web Conference, Oct. 2003, Sanibel Island, Florida, USA.
• 13: Chunhua Weng , John H. Gennari, Asynchronous collaborative writing through annotations, Proceedings of the 2004 ACM conference on Computer supported cooperative work, November 06-10, 2004, Chicago, Illinois, USA [doi>10.1145/1031607.1031705]
• 14: Anita Wilhelm , Yuri Takhteyev , Risto Sarvas , Nancy Van House , Marc Davis, Photo annotation on a camera phone, CHI '04 extended abstracts on Human factors in computing systems, April 24-29, 2004, Vienna, Austria [doi>10.1145/985921.986075]
• 15: S. H. You, S. H. Jang, Y. H. Kim, M. Hallett, S. H. Ahn, et al. (2005). Virtual reality-induced cortical reorganization and associated locomotor recovery in chronic stroke: an experimenter-blind randomized study. Stroke 36(6): 1166--1171.

INDEX TERMS

Primary Classification:
  J. Computer Applications
  J.3 LIFE AND MEDICAL SCIENCES
      Subjects: Medical information systems

Additional Classification:
  H. Information Systems
  H.1 MODELS AND PRINCIPLES
      H.1.2 User/Machine Systems
          Subjects: Human information processing; Human factors
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Interaction styles (e.g., commands, menus, forms, direct manipulation); Screen design (e.g., text, graphics, color); Auditory (non-speech) feedback

General Terms:
Design, Experimentation, Human Factors

Keywords:
analysis, annotation, archiving, biofeedback, information visualization