combinFormation: Mixed-initiative composition of image and text surrogates promotes information discovery

ABSTRACT

combinFormation is a mixed-initiative creativity support tool for searching, browsing, organizing, and integrating information. Images and text are connected to represent surrogates (enhanced bookmarks), optimizing the use of human cognitive facilities. Composition, an alternative to lists and spatial hypertext, is used to represent a collection of surrogates as a connected whole, using principles from art and design. This facilitates the creative process of information discovery, in which humans develop new ideas while finding and collecting information. To provoke the user to think about the large space of potentially relevant information resources, a generative agent proactively engages in collecting information resources, forming image and text surrogates, and composing them visually. The agent develops the collection and its visual representation over time, enabling the user to see ideas and relationships. To keep the human in control, we develop interactive mechanisms for authoring the composition and directing the agent. In a field study in an interdisciplinary course on The Design Process, over a hundred students alternated using combinFormation and Google+Word to collect prior work on information discovery invention assignments. The students that used combinFormation's mixed-initiative composition of image and text surrogates performed better.

REFERENCES

• 1: Robert Amar , John Stasko, BEST PAPER: A Knowledge Task-Based Framework for Design and Evaluation of Information Visualizations, Proceedings of the IEEE Symposium on Information Visualization, p.143-150, October 10-12, 2004 [doi>10.1109/INFOVIS.2004.10]
• 2: Baddeley, A. D. 1992. Is working memory working?, Quart. J. Exp. Psych. 44A, 1--31.
• 3: Marko Balabanovic , Yoav Shoham, Learning to surf: multiagent systems for adaptive web page recommendation, Stanford University, Stanford, CA, 1998
• 4: Michelle Q. Wang Baldonado , Terry Winograd, SenseMaker: an information-exploration interface supporting the contextual evolution of a user's interests, Proceedings of the SIGCHI conference on Human factors in computing systems, p.11-18, March 22-27, 1997, Atlanta, Georgia, United States [doi>10.1145/258549.258563]
• 5: Belkin, N. 1982. ASK for Information Retrieval: Part I. Background and Theory. J. Documentat. 38, 2, 299--304.
• 6: Eric A. Bier , Adam Perer, Icon abacus and ghost icons, Proceedings of the 5th ACM/IEEE-CS joint conference on Digital libraries, June 07-11, 2005, Denver, CO, USA [doi>10.1145/1065385.1065508]
• 7: Cynthia Breazeal, Designing Sociable Robots, MIT Press, Cambridge, MA, 2002
• 8: Burke, M. 1999. Organization of Multimedia Resources: Principles and Practice of Information Retrieval. Gower Publishing Company, Hampshire, U.K.
• 9: Edward Cutrell , Susan T. Dumais , Jaime Teevan, Searching to eliminate personal information management, Communications of the ACM, v.49 n.1, January 2006 [doi>10.1145/1107458.1107492]
• 10: Douglass R. Cutting , David R. Karger , Jan O. Pedersen , John W. Tukey, Scatter/Gather: a cluster-based approach to browsing large document collections, Proceedings of the 15th annual international ACM SIGIR conference on Research and development in information retrieval, p.318-329, June 21-24, 1992, Copenhagen, Denmark [doi>10.1145/133160.133214]
• 11: Michelangelo Diligenti , Frans Coetzee , Steve Lawrence , C. Lee Giles , Marco Gori, Focused Crawling Using Context Graphs, Proceedings of the 26th International Conference on Very Large Data Bases, p.527-534, September 10-14, 2000
• 12: Wei Ding , Gary Marchionini , Dagobert Soergel, Multimodal surrogates for video browsing, Proceedings of the fourth ACM conference on Digital libraries, p.85-93, August 11-14, 1999, Berkeley, California, United States [doi>10.1145/313238.313266]
• 13: Susan Dumais , Edward Cutrell , JJ Cadiz , Gavin Jancke , Raman Sarin , Daniel C. Robbins, Stuff I've seen: a system for personal information retrieval and re-use, Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval, July 28-August 01, 2003, Toronto, Canada [doi>10.1145/860435.860451]
• 14: Eisenstein, S. M. 1975. The Film Sense. Harcourt Brace Jovanovich, New York, NY.
• 15: Finke, R., Ward, T., and Smith, S. M. 1992. Creative Cognition. MIT Press, Cambridge, MA.
• 16: G. W. Furnas, Generalized fisheye views, Proceedings of the SIGCHI conference on Human factors in computing systems, p.16-23, April 13-17, 1986, Boston, Massachusetts, United States
• 17: George W. Furnas, A fisheye follow-up: further reflections on focus + context, Proceedings of the SIGCHI conference on Human Factors in computing systems, April 22-27, 2006, Montréal, Québec, Canada [doi>10.1145/1124772.1124921]
• 18: Glenberg, A. M. 2002. The Indexical hypothesis: Meaning from language, world, and image. In Working with Words and Images. Ablex, Westport, CT.
• 19: Glenberg, A. M. and Langston, W. E. 1992. Comprehension of illustrated text: Pictures help to build mental models. J. Mem. Lang. 31, 2, 129--151.
• 20: Hearst, M. A. and Pederson, J. O. 1996. In Proceedings of ACM IR. 76--84.
• 21: William C. Hill , James D. Hollan , Dave Wroblewski , Tim McCandless, Edit wear and read wear, Proceedings of the SIGCHI conference on Human factors in computing systems, p.3-9, May 03-07, 1992, Monterey, California, United States [doi>10.1145/142750.142751]
• 22: Horvitz, E. Uncertainty, action, and interaction. In pursuit of mixed-initiative computing. In Intelligent Systems. IEEE Computer Society Press, Los Alamitos, CA.
• 23: Huynh, D., Mazzocchi, S., and Karger, D. 2005. Piggy Bank: Experience the semantic Web inside your Web browser. In Proceedings of the 2005 International Semantic Web Conference. 413--430.
• 24: Interface Ecology Lab. 2007. combinFormation. http://ecologylab.cs.tamu.edu/combin Formation/.
• 25: Jones, W., Dumais, S., and Bruce, H. 2002. Once found, what then?: A study of ‘keeping’ behaviors in personal use of Web information. In Proceedings of the ASIST Annual Meeting. Vol. 39. 391--402.
• 26: William Jones , Ammy Jiranida Phuwanartnurak , Rajdeep Gill , Harry Bruce, Don't take my folders away!: organizing personal information to get ghings done, CHI '05 extended abstracts on Human factors in computing systems, April 02-07, 2005, Portland, OR, USA [doi>10.1145/1056808.1056952]
• 27: Diane Kelly , Jaime Teevan, Implicit feedback for inferring user preference: a bibliography, ACM SIGIR Forum, v.37 n.2, p.18-28, Fall 2003 [doi>10.1145/959258.959260]
• 28: Andruid Kerne , Ken Perlin, Collagemachine: a model of "interface ecology", New York University, New York, NY, 2001
• 29: Andruid Kerne , Madhur Khandelwal , Vikram Sundaram, Publishing evolving metadocuments on the web, Proceedings of the fourteenth ACM conference on Hypertext and hypermedia, August 26-30, 2003, Nottingham, UK [doi>10.1145/900051.900072]
• 30: Andruid Kerne , Eunyee Koh , Blake Dworaczyk , J. Michael Mistrot , Hyun Choi , Steven M. Smith , Ross Graeber , Daniel Caruso , Andrew Webb , Rodney Hill , Joel Albea, combinFormation: a mixed-initiative system for representing collections as compositions of image and text surrogates, Proceedings of the 6th ACM/IEEE-CS joint conference on Digital libraries, June 11-15, 2006, Chapel Hill, NC, USA [doi>10.1145/1141753.1141756]
• 31: Andruid Kerne , Eunyee Koh , Vikram Sundaram , J. Michael Mistrot, Generative semantic clustering in spatial hypertext, Proceedings of the 2005 ACM symposium on Document engineering, November 02-04, 2005, Bristol, United Kingdom [doi>10.1145/1096601.1096624]
• 32: Kerne, A., Mistrot, J. M., Khandelwal, M., Sundaram, V., and Koh, E. 2004a. Using composition to re-present personal collections of hypersigns. In Proceedings of Computational Semiotics in Games and New Media (CoSIGN).
• 33: Andruid Kerne , Steven M. Smith, The information discovery framework, Proceedings of the 5th conference on Designing interactive systems: processes, practices, methods, and techniques, August 01-04, 2004, Cambridge, MA, USA [doi>10.1145/1013115.1013179]
• 34: Kerne, A., Smith, S. M., Mistrot, J. M., Sundaram, V., and Khandelwal, M. 2004b. Mapping interest and design to facilitate creative process during mixed-initiative information composition. In Proceedings of Interaction: Systems, Practice and Theory, Creativity & Cognition Symposium. 1--25.
• 35: Kerne, A. and Sundaram, V. 2003. A recombinant information space. In Proceedings of the Computational Semiotics in Games and New Media (CoSIGN). 48--57.
• 36: Kerne, A., Toups, Z., Dworaczyk, B., and Khandelwal, M. 2008. Componentized XML-Java data binding framework connects software engineering with efficiency. Submitted to OOPSLA.
• 37: Madhur Khandelwal , Andruid Kerne , J. Michael Mistrot, Manipulating history in generative hypermedia, Proceedings of the fifteenth ACM conference on Hypertext and hypermedia, August 09-13, 2004, Santa Cruz, CA, USA [doi>10.1145/1012807.1012848]
• 38: Koh, E. and Kerne, A. 2006. I keep collecting: College students build and utilize collections in spite of breakdowns. In Proceedings of European Conf on Digital Libraries (ECDL). 303--314.
• 39: Lippard, L., Ed. 1971. Dadas on Art. Prentice-Hall, Englewood Cliffs, NJ.
• 40: Maier, N. R. F. 1931. Reasoning in humans ü: The solution of a problem and its appearance in consciousness. J. Compar. Physiol. Psych. 12, 181--194.
• 41: Gary Marchionini, Information seeking in electronic environments, Cambridge University Press, New York, NY, 1997
• 42: Catherine C. Marshall , Sara Bly, Saving and using encountered information: implications for electronic periodicals, Proceedings of the SIGCHI conference on Human factors in computing systems, April 02-07, 2005, Portland, Oregon, USA [doi>10.1145/1054972.1054989]
• 43: Catherine C. Marshall , William Jones, Keeping encountered information, Communications of the ACM, v.49 n.1, January 2006 [doi>10.1145/1107458.1107493]
• 44: Catherine C. Marshall , Frank M. Shipman, III , James H. Coombs, VIKI: spatial hypertext supporting emergent structure, Proceedings of the 1994 ACM European conference on Hypermedia technology, p.13-23, September 19-23, 1994, Edinburgh, Scotland [doi>10.1145/192757.192759]
• 45: Mayer, R. E. and Moreno, R. 2002. Aids to computer-based multimedia learning. Learn. Instruct. 12, 107--119.
• 46: McNee, S. M., Lam, S., Konstan, J. A., and Riedl, J. 2003. Interfaces for eliciting new user preferences in recommender systems. In Proceedings of User Modeling. 178--188.
• 47: Miller, G. A. 2006. WordNet. Princeton University Cognitive Science Laboratory, Princeton, NJ. http://wordnet.princeton.edu/.
• 48: Julie B. Morrison , Peter Pirolli , Stuart K. Card, A taxonomic analysis of what world wide web activities significantly impact people's decisions and actions, CHI '01 extended abstracts on Human factors in computing systems, March 31-April 05, 2001, Seattle, Washington [doi>10.1145/634067.634167]
• 49: Bonnie Nardi , Deborah Barreau, “Finding and reminding” revisited: appropriate metaphors for file organization at the desktop, ACM SIGCHI Bulletin, v.29 n.1, p.76-78, January 1997 [doi>10.1145/251761.248508]
• 50: Oxford English Dictionary. 1992. 2nd ed., on compact disk. Oxford University Press, Oxford, U.K.
• 51: Pirolli, P. and Card, S. K. 1999. Information foraging. Psychol. Rev. 106, 4, 643--675.
• 52: Porter, M. F. 1980. An algorithm for suffix stripping. Program, 14, 3, 130--137.
• 53: Rocchio, J. 1971. Relevance feedback in information retrieval. In The Smart Retrieval System-Experiments in Automatic Document Processing. Prentice-Hall, Englewood Cliffs, NJ, 313--323.
• 54: Gerard Salton , Michael J. McGill, Introduction to Modern Information Retrieval, McGraw-Hill, Inc., New York, NY, 1986
• 55: M. C. schraefel , Yuxiang Zhu , David Modjeska , Daniel Wigdor , Shengdong Zhao, Hunter gatherer: interaction support for the creation and management of within-web-page collections, Proceedings of the 11th international conference on World Wide Web, May 07-11, 2002, Honolulu, Hawaii, USA [doi>10.1145/511446.511469]
• 56: Frank M. Shipman , Haowei Hsieh , J. Michael Moore , Anna Zacchi, Supporting personal collections across digital libraries in spatial hypertext, Proceedings of the 4th ACM/IEEE-CS joint conference on Digital libraries, June 07-11, 2004, Tuscon, AZ, USA [doi>10.1145/996350.996433]
• 57: Ben Shneiderman , Pattie Maes, Direct manipulation vs. interface agents, interactions, v.4 n.6, p.42-61, Nov./Dec. 1997 [doi>10.1145/267505.267514]
• 58: Simon, H. A. 1971. Designing organizations for an information-rich world. In Computer, Communications, and the Public Interest. Johns Hopkins Press, Baltimore, MD, 37--72.
• 59: Smith, S. M. 1994. Getting into and out of mental ruts: A theory of fixation, incubation, and insight. In R. J. Sternberg and J. Davidson, Eds. The Nature of Insight, MIT Press, Cambridge, MA, 121--149.
• 60: Shah, J. J., Smith, S. M., Vargas-Hernandez, N., Gerkens, R., and Wulan, M. 2003. Empirical studies of design ideation: Alignment of design experiments with laboratory experiments. In Proceedings of DETC 2003: ASME International Conference on Design Theory and Methodology.
• 61: Spies, W., Ed. 2006. Max Ernst: Life and Work: An Autobiographical Collage. Thames & Hudson, London, U.K.
• 62: Spooky, Dj That Subliminal Kid. 2004. Rhythm Science. Mediawork /MIT Press, Cambridge, MA.
• 63: Sun Microsystems. 2006. Java. http://java.sun.com.
• 64: Edward Tufte, Envisioning information, Graphics Press, Cheshire, CT, 1990
• 65: Ryen W. White , Bill Kules , Steven M. Drucker , m.c. schraefel, Introduction, Communications of the ACM, v.49 n.4, April 2006 [doi>10.1145/1121949.1121978]
• 66: Barbara M. Wildemuth , Gary Marchionini , Meng Yang , Gary Geisler , Todd Wilkens , Anthony Hughes , Richard Gruss, How fast is too fast?: evaluating fast forward surrogates for digital video, Proceedings of the 3rd ACM/IEEE-CS joint conference on Digital libraries, May 27-31, 2003, Houston, Texas
• 67: Allison Woodruff , Andrew Faulring , Ruth Rosenholtz , Julie Morrsion , Peter Pirolli, Using thumbnails to search the Web, Proceedings of the SIGCHI conference on Human factors in computing systems, p.198-205, March 2001, Seattle, Washington, United States [doi>10.1145/365024.365098]
• 68: Allison Woodruff , Ruth Rosenholtz , Julie B. Morrison , Andrew Faulring , Peter Pirolli, A comparison of the use of text summaries, plain thumbnails, and enhanced thumbnails for Web search tasks, Journal of the American Society for Information Science and Technology, v.53 n.2, p.172-185, January 15, 2002 [doi>10.1002/asi.10029]
• 69: W3C. 2000. Document Object Model (DOM) Level 2 Core Specification. http://www.w3.org/TR/2000/REC-DOM-Level-2-Core-20001113/.
• 70: Yasuhiro Yamamoto , Kumiyo Nakakoji , Yoshiyuki Nishinaka , Mitsuhiro Asada , Ryouichi Matsuda, What is the space for?: the role of space in authoring hypertext representations, Proceedings of the sixteenth ACM conference on Hypertext and hypermedia, September 06-09, 2005, Salzburg, Austria [doi>10.1145/1083356.1083378]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Search process

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.7 Digital Libraries
          Subjects: User issues; Collection
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.1 Multimedia Information Systems

General Terms:
Design, Experimentation, Human Factors

Keywords:
Creativity support tools, clustering, collections, creative cognition, exploratory search, field study, focused crawler, information discovery, mixed-initiative systems, relevance feedback, semantics, software agents