Extensible middleware framework for multimodal interfaces in distributed environments

ABSTRACT

We present a framework to manage multimodal applications and interfaces in a reusable and extensible manner. We achieve this by focusing the architecture both on applications' needs and devices' capabilities. One particular domain we want to approach is collaborative environments where several modalities and applications make it necessary to provide for an extensible system combining diverse components across heterogeneous platforms on-the-fly. This paper describes the proposed framework and its main contributions in the context of an architectural application scenario. We demonstrate how to connect different non-conventional applications and input modalities around an immersive environment (tiled display wall).

REFERENCES

• 1: Araújo, B., Guerreiro, T., Jorge, J., Pereira, J., Jota, R. Leme Wall: Desenvolvendo um sistema de Multi-projecção. Proceeding of 13° EPCG , Vila Real, Portugal, Oct 2005.
• 2: Richard A. Bolt, “Put-that-there”: Voice and gesture at the graphics interface, ACM SIGGRAPH Computer Graphics, v.14 n.3, p.262-270, July 1980
• 3: Cohen, P. R., Chen, L., Clow, J., Johnston, M., McGee, D., Pittman, J., and Smith, I., 1996, Quickset: A multimodal interface for the distributed interactive simulation, in Proceedings of the ACM Symposium on User Interface Software and Technology (UIST'96), Demonstration Session, Seattle, WA.
• 4: Fritzson, Rich, Finin, Tim, McKay, Don and McEntire, Robin, 1994, KQML - A Language and Protocol for Knowledge and Information Exchange, in 13th International Distributed Artificial Intelligence Workshop, Seattle WA.
• 5: Frans Flippo , Allen Krebs , Ivan Marsic, A framework for rapid development of multimodal interfaces, Proceedings of the 5th international conference on Multimodal interfaces, November 05-07, 2003, Vancouver, British Columbia, Canada [doi>10.1145/958432.958455]
• 6: Jorge, J., Silva, N., Cardoso, T. Gides++, Proceedings of 12° EPCG, Porto, Portugal, Oct 2003.
• 7: Martin, D. L., Cheyer, A. J., Moran, and D. B., 1999, The open agent architecture: a framework for building distributed software systems, in Applied Artificial Intelligence, vol. 13, nos. 1--2, pp. 21--128.
• 8: OSGA, 2007, Open Source Groupware Architecture (April 5, 2007), http://www.osga.net.
• 9: A Real-Time Framework for Natural Multimodal Interaction with Large Screen Displays, Proceedings of the 4th IEEE International Conference on Multimodal Interfaces, p.349, October 14-16, 2002 [doi>10.1109/ICMI.2002.1167020]
• 10: Onn Shehory , Katia Sycara, The RETSINA communicator, Proceedings of the fourth international conference on Autonomous agents, p.199-200, June 03-07, 2000, Barcelona, Spain [doi>10.1145/336595.337363]
• 11: XMLBlaster, 2007, XMLBlaster (April 5, 2007); http://www.xmlblaster.org.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.3 Group and Organization Interfaces

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.1 Multimedia Information Systems
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)

General Terms:
Algorithms, Design, Human Factors

Keywords:
capabilitie, collaborative, extensible, framework, multimodal interfaces, reusable