Untangling compound documents on the web

ABSTRACT

Most text analysis is designed to deal with the concept of a "document", namely a cohesive presentation of thought on a unifying subject. By contrast, individual nodes on the World Wide Web tend to have a much smaller granularity than text documents. We claim that the notions of "document" and "web node" are not synonymous, and that authors often tend to deploy documents as collections of URLs, which we call "compound documents". In this paper we present new techniques for identifying and working with such compound documents, and the results of some large-scale studies on such web documents. The primary motivation for this work stems from the fact that information retrieval techniques are better suited to working on documents than individual hypertext nodes.

REFERENCES

• 1: HTML 4.01 specification, W3C recommendation. http://www.w3.org/TR/REC-html40/struct/links.html, December 1999.
• 2: Lada~A. Adamic and Eytan Adar. Friends and neighbors on the web. http://www.hpl.hp.com/shl/people/eytan/fnn.pdf.
• 3: Robert M. Akscyn , Donald L. McCracken , Elise A. Yoder, KMS: a distributed hypermedia system for managing knowledge in organizations, Communications of the ACM, v.31 n.7, p.820-835, July 1988 [doi>10.1145/48511.48513]
• 4: T. Berners-Lee, R. Fielding, and L. Masinter. Uniform resource identifiers (URI): Generic syntax. http://www.ietf.org/rfc/rfc2396.txt. RFC 2396.
• 5: Tim Berners-Lee, James Hendler, and Ora Lassila. The semantic web. Scientific American, May 2001. See http://www.sciam.com/2001/0501issue/0501berners-lee.html.
• 6: Rodrigo A. Botafogo , Ben Shneiderman, Identifying aggregates in hypertext structures, Proceedings of the third annual ACM conference on Hypertext, p.63-74, December 15-18, 1991, San Antonio, Texas, United States [doi>10.1145/122974.122981]
• 7: Vannevar Bush. As we may think. The Atlantic Monthly, 176(1):101--108, July 1945.
• 8: Michael Chen, Marti~A. Hearst, Jason Hong, and James Lin. Cha-cha: A system for organizing intranet search results. In USENIX Symposium on Internet Technologies and Systems, 1999.
• 9: R. C. Daley and P. G. Neumann. A general-purpose file system for secondary storage. In AFIPS Conference Proceedings, volume 27, pages 213--229, 1965. http://www.multicians.org/fjcc4.html.
• 10: Frank,G. Halasz, Reflections on NoteCards: seven issues for the next generation of hypermedia systems, Communications of the ACM, v.31 n.7, p.836-852, July 1988 [doi>10.1145/48511.48514]
• 11: Frank Halasz , Mayer Schwartz, The Dexter hypertext reference model, Communications of the ACM, v.37 n.2, p.30-39, Feb. 1994 [doi>10.1145/175235.175237]
• 12: M. M. Kessler. Bibliographic coupling between scientific papers. American Documentation, 14, 1963.
• 13: Ravi Kumar , Prabhakar Raghavan , Sridhar Rajagopalan , Andrew Tomkins, Extracting Large-Scale Knowledge Bases from the Web, Proceedings of the 25th International Conference on Very Large Data Bases, p.639-650, September 07-10, 1999
• 14: Wen-Syan Li , K. Selçuk Candan , Quoc Vu , Divyakant Agrawal, Query Relaxation by Structure and Semantics for Retrieval of Logical Web Documents, IEEE Transactions on Knowledge and Data Engineering, v.14 n.4, p.768-791, July 2002 [doi>10.1109/TKDE.2002.1019213]
• 15: Yoshiaki Mizuuchi , Keishi Tajima, Finding context paths for Web pages, Proceedings of the tenth ACM Conference on Hypertext and hypermedia : returning to our diverse roots: returning to our diverse roots, p.13-22, February 21-25, 1999, Darmstadt, Germany [doi>10.1145/294469.294474]
• 16: Theodor~Holm Nelson. Embedded markup considered harmful. http://www.xml.com/pub/a/w3j/s3.nelson.html, October 1997.
• 17: David M. Pennock, Gary W. Flake, Steve Lawrence, Eric J. Glover, and C. Les Giles. Winners don't take all: Characterizing the competition for links on the web. Proceedings of the National Academy of Science, 99(8):5207--5211, April 16 2002.
• 18: Craig Silverstein, Monika Henzinger, Hannes Marais, and Michael Moricz. Analysis of a very large altavista query log. Technical report, DEC Systems Research Center, 1998. Technical note 1998-14.
• 19: H. A. Simon. On a class of skew distribution functions. Biometrika, 42(3/4):425--440, 1955.
• 20: H. G. Small. Co-citation in the scientific literature: A new measure of the relationship between two documents. Journal of American Society for Information Science, 24(4):265--269, 1973.
• 21: Ellen Spertus, ParaSite: mining structural information on the Web, Selected papers from the sixth international conference on World Wide Web, p.1205-1215, September 1997, Santa Clara, California, United States
• 22: Amanda Spink , Dietmar Wolfram , Major B. J. Jansen , Tefko Saracevic, Searching the Web: the public and their queries, Journal of the American Society for Information Science and Technology, v.52 n.3, p.226-234, Feb. 1, 2001 [doi>10.1002/1097-4571(2000)9999:9999<::AID-ASI1591>3.3.CO;2-I]
• 23: Keishi Tajima, Kenji Hatano, Takeshi Matsukura, Ryouichi Sano, and Katsumi Tanaka. Discovery and retrieval of logical information units in web. In Proceedings of the Workshop on Organizing Web Space (WOWS 99), pages 13--23, Berkeley, CA, August 1999.
• 24: Duncan J. Watts and Steven H. Strogatz. Collective dynamics of "small-world networks". Nature, 393:440--442, June 4 1998.
• 25: Ron Weiss , Bienvenido Vélez , Mark A. Sheldon, HyPursuit: a hierarchical network search engine that exploits content-link hypertext clustering, Proceedings of the the seventh ACM conference on Hypertext, p.180-193, March 16-20, 1996, Bethesda, Maryland, United States [doi>10.1145/234828.234846]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval

General Terms:
Algorithms, Experimentation, Measurement

Keywords:
composites, hypertext, information retrieval, semantic web, wasted space