Integrating sketch and speech inputs using spatial information

ABSTRACT

Since the development of multimodal spatial query, the integration technique in determining the correct pair of multimodal inputs remains a problem in multimodal fusion. Although there exist integration techniques that have been proposed to resolve this problem, they are limited to the interaction with predefined speech and sketch commands. Furthermore, they are only designed to resolve the spatial query with single speech input and single sketch input. Therefore, when it comes to the introduction of multiple speech and sketch inputs in a single query, all the existing integration techniques are unable to resolve it. To date, no integration technique has been found that can resolve the Multiple Sentences and Sketch Objects Spatial Query. In this paper, the limitations of the existing integration techniques are discussed. A new integration technique in resolving this problem is described and compared with the widely used integration technique, Unification-based Integration Technique.

REFERENCES

• 1: Blaser, A. Geo-Spatial Sketches (A study of people's sketching behavior and strategies). Technical Report, National Center of Geographic Information and Analysis, University of Maine, Orono, ME, 1998.
• 2: Egenhofer, M. Why not SQL! International Journal of Geographic Information Systems 6(2), 86--95, 1992.
• 3: Egenhofer, M. Multi-Modal Spatial Querying. Seventh International Symposium on Spatial Data Handling (SDH '96), Delft, The Netherlands, Kraak, M.-J. and Molenaar, M. (eds.), 12B.1--15, 1996.
• 4: Michael Johnston , Philip R. Cohen , David McGee , Sharon L. Oviatt , James A. Pittman , Ira Smith, Unification-based multimodal integration, Proceedings of the eighth conference on European chapter of the Association for Computational Linguistics, p.281-288, July 07-12, 1997, Madrid, Spain
• 5: Michael Johnston, Unification-based multimodal parsing, Proceedings of the 36th annual meeting on Association for Computational Linguistics, August 10-14, 1998, Montreal, Quebec, Canada
• 6: Michael Johnston , Srinivas Bangalore , Gunaranjan Vasireddy , Amanda Stent , Patrick Ehlen , Marilyn Walker , Steve Whittaker , Preetam Maloor, MATCH: an architecture for multimodal dialogue systems, Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, July 07-12, 2002, Philadelphia, Pennsylvania [doi>10.3115/1073083.1073146]
• 7: Schlaisich, I. and Egenhofer, M. Multimodal Spatial Querying: What People Sketch and Talk About. 1st International Conference on Universal Access in Human-Computer Interaction, New Orleans, LA. Stephanidis. C. (eds.), 732--736, 2001.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)

General Terms:
Design, Experimentation, Human Factors, Measurement, Performance

Keywords:
multimodal interaction, multimodal spatial query, multimodal spatial scene description, multiple sentences and sketch objects spatial query, spatial query