Enabling context-sensitive information seeking

ABSTRACT

Information seeking is an important but often difficult task, especially when it involves large and complex data sets. We hypothesize that a context-sensitive interaction paradigm would greatly assist users in their information seeking. Such a paradigm would allow users to both express their requests and receive requested information in context. Driven by this hypothesis, we have taken rigorous steps to design, develop, and evaluate a full-fledged, context-sensitive information system. We started with a Wizard-of-OZ (WOZ) study to verify the effectiveness of our envi-sioned system. We then built a fully automated system based on the findings from our WOZ study. We targeted the development and integration of two sets of technologies: context-sensitive mul-timodal input interpretation and multimedia output generation. Finally, we formally evaluated the usability of our system in real world conditions. The results show that our system greatly improves the users' ability to perform practical information-seek-ing tasks. These results not only confirm our initial hypothesis, but they also indicate the practicality of our approaches.

REFERENCES

• 1: Christopher Ahlberg , Ben Shneiderman, Visual information seeking: tight coupling of dynamic query filters with starfield displays, Proceedings of the SIGCHI conference on Human factors in computing systems: celebrating interdependence, p.313-317, April 24-28, 1994, Boston, Massachusetts, United States [doi>10.1145/191666.191775]
• 2: James Allen , George Ferguson , Amanda Stent, An architecture for more realistic conversational systems, Proceedings of the 6th international conference on Intelligent user interfaces, p.1-8, January 14-17, 2001, Santa Fe, New Mexico, United States [doi>10.1145/359784.359822]
• 3: Elisabeth André , Thomas Rist, Generating coherent presentations employing textual and visual material, Artificial Intelligence Review, v.9 n.2-3, p.147-165, June 1995 [doi>10.1007/BF00849177]
• 4: Blaine A. Bell , Steven K. Feiner, Dynamic space management for user interfaces, Proceedings of the 13th annual ACM symposium on User interface software and technology, p.239-248, November 06-08, 2000, San Diego, California, United States [doi>10.1145/354401.354790]
• 5: Joyce Y. Chai , Pengyu Hong , Michelle X. Zhou, A probabilistic approach to reference resolution in multimodal user interfaces, Proceedings of the 9th international conference on Intelligent user interface, January 13-16, 2004, Funchal, Madeira, Portugal [doi>10.1145/964442.964457]
• 6: Context-Based Multimodal Input Understanding in Conversational Systems, Proceedings of the 4th IEEE International Conference on Multimodal Interfaces, p.87, October 14-16, 2002 [doi>10.1109/ICMI.2002.1166974]
• 7: Steven K. Feiner , Kathleen R. McKeown, Automating the Generation of Coordinated Multimedia Explanations, Computer, v.24 n.10, p.33-41, October 1991 [doi>10.1109/2.97249]
• 8: Barbara J. Grosz , Candace L. Sidner, Attention, intentions, and the structure of discourse, Computational Linguistics, v.12 n.3, p.175-204, July-September 1986
• 9: Hiroshi Hosobe, A modular geometric constraint solver for user interface applications, Proceedings of the 14th annual ACM symposium on User interface software and technology, November 11-14, 2001, Orlando, Florida [doi>10.1145/502348.502362]
• 10: K. Houck. Contextual revision in information-seeking conversation systems. In Proc. ICSLP '04, pages 201--204.
• 11: Michael Johnston , Srinivas Bangalore , Gunaranjan Vasireddy , Amanda Stent , Patrick Ehlen , Marilyn Walker , Steve Whittaker , Preetam Maloor, MATCH: an architecture for multimodal dialogue systems, Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, July 07-12, 2002, Philadelphia, Pennsylvania [doi>10.3115/1073083.1073146]
• 12: Michael Johnston , Philip R. Cohen , David McGee , Sharon L. Oviatt , James A. Pittman , Ira Smith, Unification-based multimodal integration, Proceedings of the eighth conference on European chapter of the Association for Computational Linguistics, p.281-288, July 07-12, 1997, Madrid, Spain
• 13: Stephan Kerpedjiev , Giuseppe Carenini , Steven F. Roth , Johanna D. Moore, Integrating planning and task-based design for multimedia presentation, Proceedings of the 2nd international conference on Intelligent user interfaces, p.145-152, January 06-09, 1997, Orlando, Florida, United States [doi>10.1145/238218.238316]
• 14: Gary Marchionini, Information seeking in electronic environments, Cambridge University Press, New York, NY, 1995
• 15: Mark T. Maybury, Planning multimedia explanations using communicative acts, Intelligent multimedia interfaces, American Association for Artificial Intelligence, Menlo Park, CA, 1993
• 16: Mark T. Maybury , Wolfgang Wahlster, Readings in intelligent user interfaces, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1998
• 17: Michael C. McCord, Slot Grammar: A System for Simpler Construction of Practical Natural Language Grammars, Proceedings of the International Symposium on Natural Language and Logic, p.118-145, May 09-11, 1989
• 18: Ruth C. T. Morris, Toward a user-centered information service, Journal of the American Society for Information Science, v.45 n.1, p.20-30, Jan. 1994 [doi>10.1002/(SICI)1097-4571(199401)45:1<20::AID-ASI3>3.0.CO;2-N]
• 19: J. G. Neal , C. Y. Thielman , Z. Dobes , S. M. Haller , S. C. Shapiro, Natural language with integrated deictic and graphic gestures, Readings in intelligent user interfaces, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1998
• 20: S. Oviatt. Multimodal interactive maps: Designing for human performance. Human-Computer Interaction, 12:93--129, 1997.
• 21: S. Pan. A multi-layer conversation management approach for information-seeking applications. In Proc. ICSLP '04, pages 245--248.
• 22: S. Pan and J. Shaw. SEGUE: A hybrid case-based natural language generator. In Proc. INLG '04, pages 130--140, 2004.
• 23: Shimei Pan , Siwei Shen , Michelle X. Zhou , Keith Houck, Two-way adaptation for robust input interpretation in practical multimodal conversation systems, Proceedings of the 10th international conference on Intelligent user interfaces, January 10-13, 2005, San Diego, California, USA [doi>10.1145/1040830.1040849]
• 24: Ana-Maria Popescu , Oren Etzioni , Henry Kautz, Towards a theory of natural language interfaces to databases, Proceedings of the 8th international conference on Intelligent user interfaces, January 12-15, 2003, Miami, Florida, USA [doi>10.1145/604045.604070]
• 25: Xuehua Shen , Bin Tan , ChengXiang Zhai, Context-sensitive information retrieval using implicit feedback, Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, August 15-19, 2005, Salvador, Brazil [doi>10.1145/1076034.1076045]
• 26: Jaime Teevan , Christine Alvarado , Mark S. Ackerman , David R. Karger, The perfect search engine is not enough: a study of orienteering behavior in directed search, Proceedings of the SIGCHI conference on Human factors in computing systems, p.415-422, April 24-29, 2004, Vienna, Austria [doi>10.1145/985692.985745]
• 27: Jaime Teevan , Susan T. Dumais , Eric Horvitz, Personalizing search via automated analysis of interests and activities, Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, August 15-19, 2005, Salvador, Brazil [doi>10.1145/1076034.1076111]
• 28: Zhen Wen , Michelle X. Zhou , Vikram Aggarwal, An Optimization-based Approach to Dynamic Visual Context Management, Proceedings of the Proceedings of the 2005 IEEE Symposium on Information Visualization, p.25, October 23-25, 2005 [doi>10.1109/INFOVIS.2005.6]
• 29: M. Zancanaro, O. Stock, and C. Strapparava. Multimodal interaction for information access: Exploiting cohesion. Computational Intelligence, 13(7):439--464, 1997.
• 30: Michelle X. Zhou , Vikram Aggarwal, An optimization-based approach to dynamic data content selection in intelligent multimedia interfaces, Proceedings of the 17th annual ACM symposium on User interface software and technology, October 24-27, 2004, Santa Fe, NM, USA [doi>10.1145/1029632.1029673]
• 31: M. Zhou and M. Chen. Automated generation of graphic sketches by examples. In IJCAI '03, pages 65--71.
• 32: Michelle X. Zhou , Zhen Wen , Vikram Aggarwal, A graph-matching approach to dynamic media allocation in intelligent multimedia interfaces, Proceedings of the 10th international conference on Intelligent user interfaces, January 10-13, 2005, San Diego, California, USA [doi>10.1145/1040830.1040861]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: User-centered design

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.1 Multimedia Information Systems

General Terms:
Design, Experimentation, Human Factors, Languages