Take a look at the new version of this page: [ beta version ]. Tell us what you think.

ABSTRACT

The invention of the hyperlink and the HTTP transmission protocol caused an amazing new structure to appear on the Internet -- the World Wide Web. With the Web, there came spiders, robots, and Web crawlers, which go from one link to the next checking Web health, ferreting out information and resources, and imposing organization on the huge collection of information (and dross) residing on the net. This paper reports on the use of one such crawler to synthesize document collections on various topics in science, mathematics, engineering and technology. Such collections could be part of a digital library.

REFERENCES

• 1: W. Arms. Automated digital libraries: How effectively can computers be used for the skill tasks of professional librarianship. D-Lib Magazine: The Magazine of Digital Library Research, July 2000. http://www.dlib.org/dlib/july00/arms/07arms.html
• 2: Arvind Arasu , Junghoo Cho , Hector Garcia-Molina , Andreas Paepcke , Sriram Raghavan, Searching the Web, ACM Transactions on Internet Technology (TOIT), v.1 n.1, p.2-43, Aug. 2001 [doi>10.1145/383034.383035]
• 3: R. K. Belew. Finding Out About. Cambridge Press, 2001
• 4: Israel Ben-Shaul , Michael Herscovici , Michal Jacovi , Yoelle S. Maarek , Dan Pelleg , Menachem Shtalhaim , Vladimir Soroka , Sigalit Ur, Adding support for dynamic and focused search with Fetuccino, Proceeding of the eighth international conference on World Wide Web, p.1653-1665, May 1999, Toronto, Canada
• 5: Krishna Bharat , Monika R. Henzinger, Improved algorithms for topic distillation in a hyperlinked environment, Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, p.104-111, August 24-28, 1998, Melbourne, Australia [doi>10.1145/290941.290972]
• 6: C. M. Bowman, P. B. Danzig, D. R. Hardy, U. Manber, and M. F. Schwartz. Harvest: A scalable, customizable discovery and access system. Technical Report CU-CS-732-94, Department of Computer Science, University of Colorado, Boulder, July 1994
• 7: C. M. Bowman, P. B. Danzig, D. R. Hardy, U. Manber, and M. F. Schwartz. The Harvest information discovery and access system, 1994. Additional information available http://archive.ncsa.uiuc.edu/SDG/IT94/Proceedings/Searching/schwartz.harvest/schwartz.harvest.html
• 8: Sergey Brin , Lawrence Page, The anatomy of a large-scale hypertextual Web search engine, Proceedings of the seventh international conference on World Wide Web 7, p.107-117, April 1998, Brisbane, Australia
• 9: A. Broder, S. Glassman, and M. Manasse. Clustering the Web, 1999. Available: http://www.research.compaq.com/SRC/articles/199707/cluster.html
• 10: Andrei Z. Broder , Steven C. Glassman , Mark S. Manasse , Geoffrey Zweig, Syntactic clustering of the Web, Selected papers from the sixth international conference on World Wide Web, p.1157-1166, September 1997, Santa Clara, California, United States
• 11: S. Jeromy Carrière , Rick Kazman, WebQuery: searching and visualizing the Web through connectivity, Selected papers from the sixth international conference on World Wide Web, p.1257-1267, September 1997, Santa Clara, California, United States
• 12: Soumen Chakrabarti, Recent results in automatic Web resource discovery, ACM Computing Surveys (CSUR), v.31 n.4es, Dec. 1999 [doi>10.1145/345966.346007]
• 13: Soumen Chakrabarti , Byron Dom , Prabhakar Raghavan , Sridhar Rajagopalan , David Gibson , Jon Kleinberg, Automatic resource compilation by analyzing hyperlink structure and associated text, Proceedings of the seventh international conference on World Wide Web 7, p.65-74, April 1998, Brisbane, Australia
• 14: S. Chakrabarti, B. E. Dom, D. Gibson, R. Kumar, P. Raghavan, S. Rajagopalan, and A. Tomkins. Experiments in topic distillation. In Proceedings of the ACM SIGIR Workshop on Hypertext Information Retrieval on the Web, Melbourne, Australia, 1998. ACM. Available: http://www.almaden.ibm.com/cs/k53/abstract.html
• 15: Soumen Chakrabarti , Martin van den Berg , Byron Dom, Focused crawling: a new approach to topic-specific Web resource discovery, Proceeding of the eighth international conference on World Wide Web, p.1623-1640, May 1999, Toronto, Canada
• 16: C. Chekuri, M. Goldwasser, P. Raghavan, and E. Upfal. Web search using automatic classification, 1997. Available at http://cm.bell-labs.com/who/chekuri/postscript/web.ps.gz Current as of December 5, 2001
• 17: Fah-Chun Cheong, Internet agents: spiders, wanderers, brokers, and bots, New Riders Publishing, Indianapolis, IN, 1996
• 18: Brian D. Davison, Topical locality in the Web, Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, p.272-279, July 24-28, 2000, Athens, Greece [doi>10.1145/345508.345597]
• 19: Jeffrey Dean , Monika R. Henzinger, Finding related pages in the World Wide Web, Proceeding of the eighth international conference on World Wide Web, p.1467-1479, May 1999, Toronto, Canada
• 20: Michelangelo Diligenti , Frans Coetzee , Steve Lawrence , C. Lee Giles , Marco Gori, Focused Crawling Using Context Graphs, Proceedings of the 26th International Conference on Very Large Data Bases, p.527-534, September 10-14, 2000
• 21: E. Garfield. Mapping the structure of science, pages 98--147. John Wiley & Sons, Inc. NY, 1979. Available at http://www.garfield.library.upenn.edu/ci/chapter8.pdf
• 22: David Gibson , Jon Kleinberg , Prabhakar Raghavan, Inferring Web communities from link topology, Proceedings of the ninth ACM conference on Hypertext and hypermedia : links, objects, time and space---structure in hypermedia systems: links, objects, time and space---structure in hypermedia systems, p.225-234, June 20-24, 1998, Pittsburgh, Pennsylvania, United States [doi>10.1145/276627.276652]
• 23: E.-H. S. Han and G. Karypis. Centroid-based document classification: Analysis & experimental results. Technical Report 00-017, Computer Science, University of Minnesota, Mar. 2000
• 24: T. H. Haveliwala, A. Gionis, and P. Indyk. Scalable techniques for clustering the Web. In WebDB'2000: Third International Workshop on the Web and Databases, May 2000. Available http://www.research.att.com/conf/webdb2000/PAPERS/8c.ps
• 25: Monika R. Henzinger , Allan Heydon , Michael Mitzenmacher , Marc Najork, On near-uniform URL sampling, Proceedings of the 9th international World Wide Web conference on Computer networks : the international journal of computer and telecommunications netowrking, p.295-308, June 2000, Amsterdam, The Netherlands
• 26: Monika R. Henzinger , Allan Heydon , Michael Mitzenmacher , Marc Najork, Measuring index quality using random walks on the Web, Proceeding of the eighth international conference on World Wide Web, p.1291-1303, May 1999, Toronto, Canada
• 27: Allan Heydon , Marc Najork, Mercator: A scalable, extensible Web crawler, World Wide Web, v.2 n.4, p.219-229, 1999 [doi>10.1023/A:1019213109274]
• 28: Jon M. Kleinberg, Hubs, authorities, and communities, ACM Computing Surveys (CSUR), v.31 n.4es, Dec. 1999 [doi>10.1145/345966.345982]
• 29: V. Kluev, Compiling document collections from the Internet, ACM SIGIR Forum, v.34 n.2, p.9-14, Fall 2000 [doi>10.1145/381258.381264]
• 30: S. Lawrence and C. L. Giles. Accessibility of information on the Web. Nature, 400(8), July 1999
• 31: R. Lempel , S. Moran, SALSA: the stochastic approach for link-structure analysis, ACM Transactions on Information Systems (TOIS), v.19 n.2, p.131-160, April 2001 [doi>10.1145/382979.383041]
• 32: Andrew Kachites McCallum , Kamal Nigam , Jason Rennie , Kristie Seymore, Automating the Construction of Internet Portals with Machine Learning, Information Retrieval, v.3 n.2, p.127-163, July 2000 [doi>10.1023/A:1009953814988]
• 33: F. Menczer and R. K. Belew. Adaptive Retrieval Agents: Internalizing Local Context and Scaling up to the Web, pages 1--45. 1999
• 34: Sougata Mukherjea, Organizing topic-specific web information, Proceedings of the eleventh ACM on Hypertext and hypermedia, p.133-141, May 30-June 03, 2000, San Antonio, Texas, United States [doi>10.1145/336296.336346]
• 35: Sougata Mukherjea, WTMS: a system for collecting for collecting and analyzing topic-specific Web information, Proceedings of the 9th international World Wide Web conference on Computer networks : the international journal of computer and telecommunications netowrking, p.457-471, June 2000, Amsterdam, The Netherlands
• 36: Peter Mutschke, Enhancing Information Retrieval in Federated Bibliographic Data Sources Using Author Network Based Stratagems, Proceedings of the 5th European Conference on Research and Advanced Technology for Digital Libraries, p.287-299, September 04-09, 2001
• 37: M. Najork and A. Heydon. High-performance Web crawling. Technical Report Research Report 173, Compaq SRC, Sept. 2001. Available at http://gatekeeper.research.compaq.com/pub/DEC/SRC/research-reports/abstracts/src-rr-173.html
• 38: Marc Najork , Janet L. Wiener, Breadth-first crawling yields high-quality pages, Proceedings of the 10th international conference on World Wide Web, p.114-118, May 01-05, 2001, Hong Kong, Hong Kong [doi>10.1145/371920.371965]
• 39: Gerard. Salton, Automatic Information Organization and Retrieval., McGraw Hill Text, 1968
• 40: B. Saulnier. Portal power. Cornell Engineering Magazine, pages 16--21, Fall 2001. Available:http://www.engineering.cornell.edu/engrMagazine/
• 41: Raymie Stata , Krishna Bharat , Farzin Maghoul, The term vector database: fast access to indexing terms for Web pages, Proceedings of the 9th international World Wide Web conference on Computer networks : the international journal of computer and telecommunications netowrking, p.247-255, June 2000, Amsterdam, The Netherlands
• 42: D. Voss. Better searching through science. Science, 293(5537):2024, 2001. Available: http://www.sciencemag.org/cgi/content/full/293/5537/2024
• 43: Peter Willett, Recent trends in hierarchic document clustering: a critical review, Information Processing and Management: an International Journal, v.24 n.5, p.577-597, 1988 [doi>10.1016/0306-4573(88)90027-1]
• 44: Ian H. Witten , David Bainbridge , Stefan J. Boddie, Power to the people: end-user building of digital library collections, Proceedings of the 1st ACM/IEEE-CS joint conference on Digital libraries, p.94-103, January 2001, Roanoke, Virginia, United States [doi>10.1145/379437.379458]
• 45: Ian H. Witten , Stefan J. Boddie , David Bainbridge , Rodger J. McNab, Greenstone: a comprehensive open-source digital library software system, Proceedings of the fifth ACM conference on Digital libraries, p.113-121, June 02-07, 2000, San Antonio, Texas, United States [doi>10.1145/336597.336650]
• 46: Oren Zamir , Oren Etzioni, Web document clustering: a feasibility demonstration, Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, p.46-54, August 24-28, 1998, Melbourne, Australia [doi>10.1145/290941.290956]
• 47: L. L. Zia. The NSF national science, technology, engineering, and mathematics education digital library (NSDL) program: New projects and a project report. D-Lib Magazine: The Magazine of Digital Library Research, 7(11), Nov. 2001

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
        D.2.12 Interoperability

Additional Classification:
  H. Information Systems
  H.1 MODELS AND PRINCIPLES
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.1 Content Analysis and Indexing
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.4 Hypertext/Hypermedia

General Terms:
Design, Experimentation, Performance

Keywords:
NSDL, World Wide Web, crawling, information retrieval clustering, mercator, topic management