Deriving image-text document surrogates to optimize cognition

ABSTRACT

The representation of information collections needs to be optimized for human cognition. While documents often include rich visual components, collections, including personal collections and those generated by search engines, are typically represented by lists of text-only surrogates. By concurrently invoking complementary components of human cognition, combined image-text surrogates will help people to more effectively see, understand, think about, and remember an information collection. This research develops algorithmic methods that use the structural context of images in HTML documents to associate meaningful text and thus derive combined image-text surrogates. Our algorithm first recognizes which documents consist essentially of informative and multimedia content. Then, the algorithm recognizes the informative sub-trees within each such document, discards advertisements and navigation, and extracts images with contextual descriptions. Experimental results demonstrate the algorithm's efficacy. An implementation of the algorithm is provided in combinFormation, a creativity support tool for collection authoring. The enhanced image-text surrogates enhance the experiences of users finding and collecting information as part of developing new ideas.

REFERENCES

• 1: Arasu, A., Garcia-Molina, H., Extracting Structured Data from Web Pages, Proceedings of SIGMOD 2003, 337--348.
• 2: Burke, M., Organization of Multimedia Resources, Hampshire, UK: Gower, 1999.
• 3: Cai, D., He, X., Li, Z., Ma, W-Y., Wen, J-R., Hierarchical Clustering of WWW Image Search Results using Visual, Textual and Link Information, Proc Multimedia 2004, pp. 952--959.
• 4: Cai, D., Yu, S., Wen, J-R. & Ma, W-Y., VIPS: a Vision-based Page Segmentation Algorithm, Microsoft Technical Report, MSR-TR-2003-79, 2003.
• 5: Carney, R.,N., Levin, J.R., Pictorial Illustrations Still Improve Students' Learning From Text, Educational Psychology Review 14:1, March 2002.
• 6: Chang, C., Kayed, M., Girgis, M. R., Shaalan, K. F., A Survey of Web Information Extraction Systems. IEEE TKDE 18(10), 2006.
• 7: Chang, C., Lui, S., IEPAD: Information Extraction Based on Pattern Discovery, Proc WWW 2001, 681--688.
• 8: Crescenzi, V., Mecca, G., & Merialdo, P. RoadRunner: Towards Automatic Data Extraction from Large Web Sites, Proc VLDB Conference, Very Large Data Bases 2001, 109--118.
• 9: Chen, J., Zhou, B., Shi, J., Zhang, H., Fengwu, Q., Function-based object model towards website adaptation, Proc WWW 2001, 587--596.
• 10: Ding, W., Marchionini, G., Soergel, D., Multimodal Surrogates for Video Browsing, Proc JCDL 1999, 85--93.
• 11: Feng, H., Shi, R., Chua, T-S., A bootstrapping framework for annotating and retrieving WWW images, Proc Multimedia 2004, pp. 960--967.
• 12: Glenberg, A.M., Langston, W.E., Comprehension of illustrated text: Pictures help to build mental models, Journal of Memory & Language, 31(2):129--151, April 1992.
• 13: He, X., Cai, D., Wen, J-R, Ma, W-Y, Zhang, H-J, Clustering and searching WWW images using link and page layout analysis, ACM TOMCCAP, 3(2):Article No. 10, May 2007.
• 14: Interface Ecology Lab, combinFormation, http://ecologylab.cs.tamu.edu/combinFormation, last visited 01/28/2009.
• 15: Jaimes, A., Gatica-Perez, D., Sebe, N., Huang, T.S., Human-Centered Computing: Toward a Human Revolution, IEEE Computer 40(5), 2007, pp. 30--34.
• 16: Kerne, A., Koh, E., Smith, S.M., Webb, A., Dworaczyk, B., combinFormation: Mixed-Initiative Composition of Image and Text Surrogates Promotes Information Discovery, ACM TOIS, 27(1) Article No. 5, December 2008.
• 17: Koh, E., Kerne, A., Berry, S., Test Collection Management and Labeling System, Proc ACM Document Engineering 2009.
• 18: Koh, E., Caruso, D., Kerne, A., Gutierrez-Osuna, R., Elimination of Junk Document Surrogate Candidates through Pattern Recognition, Proc ACM Document Engineering 2007, 187--195.
• 19: Koh, E., Kerne, A., Webb, A., Damaraju, S., Sturdivant, D., Generating Views of the Buzz: Browsing Popular Media and Authoring using Mixed-Initiative Composition, Proc Multimedia 2007, 228--237.
• 20: Laender, A., Ribeiro-Neto, B., da Silva, A., Teixeira, J., A Brief Survey of Web Data Extraction Tools. ACM SIGMOD Record, 31(2), 2002.
• 21: Li, Z., Shi, S., Zhang, L., Improving Relevance Judgment of Web Search Results with Image Excerpts, WWW 2008, 21--30.
• 22: Liesaputra, V., Ian, W.H., Seeking information in realistic books: a user study, Proc JCDL 2008, 29--38.
• 23: Lin, S., Ho, J., Discovering Informative Content Blocks from Web Documents, Proceedings of SIGKDD 2002, 588--593.
• 24: Liu, B., Grossman, R., & Zhai, Y., Mining Data Records in Web Pages, Proc ACM SIGKDD Special Interest Group on Knowledge Discovery and Data Mining, 601--606, 2003.
• 25: Liu, Y., Zhang, D., Lu, G., Ma, W-Y., A survey of content-based image retrieval with high-level semantics, Pattern Recognition 40(1) 2007, pp. 262--282.
• 26: Mayer, R.E., Moreno, R., Animation as an Aid to Multimedia Learning, Educational Psychology Review, 14:1, March 2002.
• 27: National Science Foundation, Discoveries, http://www.nsf.gov/discoveries/, last visited 01/28/2009
• 28: New York Public Library, Digital Collections, http://www.nypl.org/digital/collections_images.html, last visited 01/28/2009.
• 29: Paivio, A., Mental Representations: a Dual Coding Approach, Oxford, England: Oxford University Press, 1986.
• 30: Reis, D. C., Golgher P. B., Silva, A. S., & Launder, A. H. F., Automatic Web News Extraction Using Tree Edit Distance, Proceedings of WWW 2004, 502--511.
• 31: Song, R., Liu, H., Wen, J-R., & Ma, W-Y., Learning Important Models for Web Page Blocks based on Layout and Content Analysis, Special Interest Group on Knowledge Discovery and Data Mining (SIGKDD) Explorations, 6(2), 14--23, 2004.
• 32: W3C, Document Object Model (DOM) Level 2 Core Specification, http://www.w3.org/TR/2000/REC-DOM-Level-2-Core-20001113/, 2000.
• 33: Wang, J., & Lochovsky, F. H., Data Extraction and Label Assignment for Web Databases, Proceedings of WWW 2003, 187--196, 2003.
• 34: Wildemuth, B., Marchionini, G., Yang, M., Geisler, G., Wilkens, T., Hughes, A., Gruss, R., How Fast Is Too Fast? Evaluating Fast Forward Surrogates for Digital Video, Proceedings of ACM/IEEE Joint Conference on Digital Libraries 2003, 221--230.
• 35: Woodruff, A., Faulring, A., Rosenholtz, R., Morrison, J., Pirolli, P., Using Thumbnails to Search the Web, Proceedings of SIGCHI 2001, 198--205.
• 36: Yi, L., Liu, B., Li, X., Eliminating Noisy Information in Web Pages for Data Mining, Proceedings of SIGKDD 2003, 296--305.
• 37: Yu, S., Cai, D., Wen, J-R., Ma, W-Y., Improving Pseudo-Relevance Feedback in Web Information Retrieval Using Web Page Segmentation, Proceedings of WWW 2003, 11--18.
• 38: Zhao, H., Meng, W., Yu, C., Mining Templates from Search Result Records of Search Engines, Proceedings of SIGKDD 2007, 884--893.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Selection process

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.4 Hypertext/Hypermedia
          Subjects: Navigation

General Terms:
Algorithms, Design, Experimentation, Performance

Keywords:
information extraction, search representation, surrogates