Smartweb: multimodal web services on the road

ABSTRACT

SmartWeb provides a context-aware user interface to web services, so that it can support the mobile user in different roles, e.g. as a car driver, a motorbiker, or a pedestrian. It provides a symmetric multimodal dialogue system [2] combining speech, gesture, haptic and video input with speech, haptic, video and acoustic output. It goes beyond traditional keyword search engines like Google by delivering higher quality results that are adapted to the mobile user's current task and situation. In mobile situations, users don't want to deal with hypertext lists of retrieved webpages, but simply want an answer to their query. If a desperate driver with a crying and acutely ill child on the backseat asks SmartWeb "Who is the closest paediatrician?" he needs just the name and address of the doctor. Based on SmartWeb's ability to combine various web services, the driver can then ask SmartWeb a follow-up question about route guidance to the doctor's practice. One of the innovative features of SmartWeb is that the user can specify whether he wants a textual or pictorial answer, a video clip or a sound file as a query result.

SmartWeb [1] provides not only an open-domain question answering machine but a multimodal web service interface for coherent dialogue, where questions and commands are interpreted according to the context of the previous conversation. For example, if the driver of our Mercedes-Benz R-Class test car asks SmartWeb "Where is the closest Italian restaurant", it will access a web service to find an appropriate restaurant and show its location on a digital map presented on the large dashboard display. The user may continue his dialog with a command like "Please guide me there with a refuelling stop at the lowest price gas station". In this case, SmartWeb combines a navigation service with a special web service that finds low gas prices. SmartWeb includes plan-based composition methods for semantic web services, so that complex tasks can be carried out for the mobile user.

One version of SmartWeb has been deployed on a BMW motorbike R1200RT, using a swivel with force feedback integrated in the handle bar. Similar to the control knob known from the iDrive interface of BMW automobiles, the biker can rotate the swivel or push it right or left in order to browse through menus or select items displayed by SmartWeb on the large high-resolution screen in the middle of the cockpit. In combination with these pointing actions, the biker can use speech input over the microphone integrated in a Bluetooth helmet to interact with SmartWeb. The multimodal dialogue system combines visual displays with speech and earcons over the speakers integrated in the helmet and haptic force feedback for output generation. For example, the biker can ask for weather forecasts along his planned route. SmartWeb accesses location-based web services via the bike's 3G wireless connection to retrieve the relevant weather forecasts. In addition, SmartWeb exploits ad-hoc Wifi connections for vehicle-to-vehicle communication based on a local danger warning ontology so that the motorbike driver can be informed of a danger ahead by a car in front of him. For example, a car detecting a large wedge of water under its wheels will pass the information wirelessly to the bike following it and SmartWeb will generate the warning "Attention! Risk of aquaplaning 100 meters ahead" using the GPS coordinates of both vehicles to compute the distance to the upcoming dangerous area. Another distinguishing feature of SmartWeb is the generation of adaptive multimodal presentations taking into account the predicted cognitive load of the biker depending on the driving speed and other factors.

This keynote presents the anatomy of SmartWeb, its ontology-based information extraction and web service composition technology and explains the distinguishing features of its multimodal dialogue and answer engine.

REFERENCES

• 1: Sonntag, D., Engel, R., Herzog, G., Pfalzgraf, A., Pfleger, N., Romanelli, M., Reithinger, N.: SmartWeb Handheld: Multimodal Interaction with Ontological Knowledge Bases and Semantic Web Services. In Huang, Th., Nijholt, A., Pantic, M., Pentland, A. (eds.): Artificial Intelligence for Human Computing. Springer, Heidelberg, 2007, 272--295.
• 2: Wolfgang Wahlster, SmartKom: Foundations of Multimodal Dialogue Systems (Cognitive Technologies), Springer-Verlag New York, Inc., Secaucus, NJ, 2006

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Voice I/O

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Natural language; Haptic I/O

General Terms:
Design, Human Factors

Keywords:
in-car spoken dialogue system, multimodal web services, vehicle-to-vehicle communication