Take a look at the new version of this page: [ beta version ]. Tell us what you think.

ABSTRACT

The increasing need to access information everywhere and at any time leads us to believe that future user interfaces, through which users interact with pervasive computing systems, must address both device and modality independence. The pervasive computing paradigm sees almost every object in the everyday environment as a system able to communicate with users and other systems in their own languages. The interaction between users and systems is therefore typically multimodal. The main challenge of multimodal interaction, that is also the main topic of this paper, lies in developing a framework that is able to process information derived from whatever input modalities, giving these inputs an appropriate representation and integrating these individual representations into a joint semantic interpretation. A description of this multimodal pervasive framework will be given in this paper, along with some details of its application in Ambient Assisted Living and the usability test that was implemented to validate its effectiveness.

REFERENCES

• 1: Michael Johnston , Srinivas Bangalore, Finite-state multimodal integration and understanding, Natural Language Engineering, v.11 n.2, p.159-187, June 2005 [doi>10.1017/S1351324904003572]
• 2: Sanjeev Kumar , Philip R. Cohen , Rachel Coulston, Multimodal interaction under exerted conditions in a natural field setting, Proceedings of the 6th international conference on Multimodal interfaces, October 13-15, 2004, State College, PA, USA [doi>10.1145/1027933.1027971]
• 3: Wolfgang Mueller , Robbie Schaefer , Steffen Bleul, Interactive Multimodal User Interfaces for Mobile Devices, Proceedings of the Proceedings of the 37th Annual Hawaii International Conference on System Sciences (HICSS'04) - Track 9, p.90286.1, January 05-08, 2004
• 4: Schatz, R., Simon, R., Anegg, H., Wegscheider, F., Niklfeld, G.: Developing Mobile Multimodal Applications, HCI 2005, Edinburgh, Scotland, September 5--7. (2005).
• 5: The Rogue Project, http://www.cs.uta.fi/~rogue/index.html
• 6: Michael Johnston , Srinivas Bangalore, Finite-state multimodal parsing and understanding, Proceedings of the 18th conference on Computational linguistics, p.369-375, July 31-August 04, 2000, Saarbrücken, Germany [doi>10.3115/990820.990874]
• 7: Yong Sun , Fang Chen , Yu (David) Shi , Vera Chung, A novel method for multi-sensory data fusion in multimodal human computer interaction, Proceedings of the 18th Australia conference on Computer-Human Interaction: Design: Activities, Artefacts and Environments, November 20-24, 2006, Sydney, Australia [doi>10.1145/1228175.1228257]
• 8: Bob Carpenter, The logic of typed feature structures, Cambridge University Press, New York, NY, 1992
• 9: Frank Rudzicz, Clavius: bi-directional parsing for generic multimodal interaction, Proceedings of the 21st International Conference on computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop, July 20-20, 2006, Sydney, Australia
• 10: D'Ulizia, A., Ferri, F., Grifoni, P.: A Hybrid Grammar-Based Approach to Multimodal Languages Specification. OTM Workshops (1), 367--376. (2007).
• 11: Chande, H.: Esperanza de vida y expectativas de salud en la edad avanzada. (2001). http://www.redadultosmayores.com
• 12: Fischer, S, Stewart, I. E., Mehta, S., Wax, R., Lapinsky, S. E.: Handheld computing in medicine. J Am Med Inform Assoc, 10(2): 139--149. (2003).

INDEX TERMS

Primary Classification:
  D. Software
  D.3 PROGRAMMING LANGUAGES
      D.3.3 Language Constructs and Features
          Subjects: Frameworks

Additional Classification:
  D. Software
  D.3 PROGRAMMING LANGUAGES
      D.3.3 Language Constructs and Features
          Subjects: Input/output

  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Input devices and strategies (e.g., mouse, touchscreen); User interface management systems (UIMS)

  J. Computer Applications
  J.3 LIFE AND MEDICAL SCIENCES
      Subjects: Health

General Terms:
Experimentation, Languages

Keywords:
ambient assisted living, framework architecture, multimodal languages, pervasive systems