Indexing and retrieval of scientific literature

ABSTRACT

The web has greatly improved access to scientific literature. However, scientific articles on the web are largely disorganized, with research articles being spread across archive sites, institution sites, journal sites, and researcher homepages. No index covers all of the available literature, and the major web search engines typically do not index the content of Postscript/PDF documents at all. This paper discusses the creation of digital libraries of scientific literature on the web, including the efficient location of articles, full-text indexing of the articles, autonomous citation indexing, information extraction, display of query-sensitive summaries and citation context, hubs and authorities computation, similar document detection, user profiling, distributed error correction, graph analysis, and detection of overlapping documents. The software for the system is available at no cost for non-commercial use.

REFERENCES

• 1: J.M. Barrie and D.E. Presti. The World Wide Web as an instructional tool. Science, 274:371-372, 1996.
• 2: Krishna Bharat , Monika R. Henzinger, Improved algorithms for topic distillation in a hyperlinked environment, Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, p.104-111, August 24-28, 1998, Melbourne, Australia [doi>10.1145/290941.290972]
• 3: Kurt D. Bollacker , Steve Lawrence , C. Lee Giles, CiteSeer: an autonous Web agent for automatic retrieval and identification of interesting publications, Proceedings of the second international conference on Autonomous agents, p.116-123, May 10-13, 1998, Minneapolis, Minnesota, United States [doi>10.1145/280765.280786]
• 4: Kurt D. Bollacker , Steve Lawrence , C. Lee Giles, A system for automatic personalized tracking of scientific literature on the Web, Proceedings of the fourth ACM conference on Digital libraries, p.105-113, August 11-14, 1999, Berkeley, California, United States [doi>10.1145/313238.313270]
• 5: Sergey Brin , James Davis , Héctor García-Molina, Copy detection mechanisms for digital documents, Proceedings of the 1995 ACM SIGMOD international conference on Management of data, p.398-409, May 22-25, 1995, San Jose, California, United States
• 6: Sergey Brin , Lawrence Page, The anatomy of a large-scale hypertextual Web search engine, Proceedings of the seventh international conference on World Wide Web 7, p.107-117, April 1998, Brisbane, Australia
• 7: Andrei Z. Broder , Steven C. Glassman , Mark S. Manasse , Geoffrey Zweig, Syntactic clustering of the Web, Selected papers from the sixth international conference on World Wide Web, p.1157-1166, September 1997, Santa Clara, California, United States
• 8: Eric W. Brown , James P. Callan , W. Bruce Croft, Fast Incremental Indexing for Full-Text Information Retrieval, Proceedings of the 20th International Conference on Very Large Data Bases, p.192-202, September 12-15, 1994
• 9: Robert D. Cameron. A universal citation database as a catalyst for reform in scholarly communication. First Monday, 2(4), 1997.
• 10: Junghoo Cho , Hector Garcia-Molina , Lawrence Page, Efficient crawling through URL ordering, Proceedings of the seventh international conference on World Wide Web 7, p.161-172, April 1998, Brisbane, Australia
• 11: D. Cutting , J. Pedersen, Optimization for dynamic inverted index maintenance, Proceedings of the 13th annual international ACM SIGIR conference on Research and development in information retrieval, p.405-411, September 05-07, 1990, Brussels, Belgium [doi>10.1145/96749.98245]
• 12: Eugene Garfield. Citation Indexing: Its Theory and Application in Science, Technology, and Humanities. Wiley, New York, 1979.
• 13: C. Lee Giles , Kurt D. Bollacker , Steve Lawrence, CiteSeer: an automatic citation indexing system, Proceedings of the third ACM conference on Digital libraries, p.89-98, June 23-26, 1998, Pittsburgh, Pennsylvania, United States [doi>10.1145/276675.276685]
• 14: Paul Ginsparg, First steps towards electronic research communication, Computers in Physics, v.8 n.4, p.390-396, July/Aug. 1994
• 15: S. Hitchcock , L. Carr , S. Harris , J. M. N. Hey , W. Hall, Citation linking: improving access to online journals, Proceedings of the second ACM international conference on Digital libraries, p.115-122, July 23-26, 1997, Philadelphia, Pennsylvania, United States [doi>10.1145/263690.263804]
• 16: Henry Kautz , Bart Selman , Mehul Shah, Referral Web: combining social networks and collaborative filtering, Communications of the ACM, v.40 n.3, p.63-65, March 1997 [doi>10.1145/245108.245123]
• 17: Jon M. Kleinberg, Authoritative sources in a hyperlinked environment, Proceedings of the ninth annual ACM-SIAM symposium on Discrete algorithms, p.668-677, January 25-27, 1998, San Francisco, California, United States
• 18: Steve Lawrence , Kurt Bollacker , C. Lee Giles, Distributed error correction, Proceedings of the fourth ACM conference on Digital libraries, p.232, August 11-14, 1999, Berkeley, California, United States [doi>10.1145/313238.313390]
• 19: Steve Lawrence , C. Lee Giles, Context and Page Analysis for Improved Web Search, IEEE Internet Computing, v.2 n.4, p.38-46, July 1998 [doi>10.1109/4236.707689]
• 20: Steve Lawrence and C. Lee Giles. Searching the World Wide Web. Science, 280(5360):98-t00, 1998.
• 21: Steve Lawrence and C. Lee Giles. Accessibility of information on the web. Nature, 400(6740):107-109, 1999.
• 22: L. Page, S. Brin, R. Motwani, and T. Winograd. The PageRank citation ranking: Bringing order to the web. 1998.
• 23: Jason Rennie , Andrew McCallum, Using Reinforcement Learning to Spider the Web Efficiently, Proceedings of the Sixteenth International Conference on Machine Learning, p.335-343, June 27-30, 1999
• 24: E. Selberg and O. Etzioni. Multi-service search and comparison using the MetaCrawler. In Proceedings of the 1995 World Wide Web Conference, 1995.
• 25: Kristie Seymore, Andrew McCallum, and Roni Rosen~ feld. Learning hidden Markov model structure for information extraction. In AAAI 99 Workshop on Machine Learning for Information Extraction, 1999.
• 26: N. Shivakumar and H. Garcia-Molina. SCAM: A copy detection mechanism for digital documents. In 2nd International Conference on the Theory and Practice of Digital Libraries, 1995.
• 27: Anthony Tomasic , Héctor García-Molina , Kurt Shoens, Incremental updates of inverted lists for text document retrieval, Proceedings of the 1994 ACM SIGMOD international conference on Management of data, p.289-300, May 24-27, 1994, Minneapolis, Minnesota, United States
• 28: Anastasios Tombros , Mark Sanderson, Advantages of query biased summaries in information retrieval, Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, p.2-10, August 24-28, 1998, Melbourne, Australia [doi>10.1145/290941.290947]
• 29: Anastasios Tombros. Reflecting User Information Needs Through Query Biased Summaries. PhD thesis, Department of Computer Science, University of Glasgow, September 1997.
• 30: Ian H. Witten , Alistair Moffat , Timothy C. Bell, Managing gigabytes (2nd ed.): compressing and indexing documents and images, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1999
• 31: I.H. Witten, C.G. Nevill-Manning, and S.J.Cunningham. Building a digital library for computer science research: technical issues. In Proceedings Australasian Computer Science Conference, Melbourne, Australia, January 1996.
• 32: I.H. Witten, C.G. Nevill-Manning, and S.J. Cunningham. Digital libraries based on fulltext retrieval. In Proceedings of WebNet 96, San Francisco, October 1996.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.1 Content Analysis and Indexing
          Subjects: Indexing methods

Additional Classification:
  H. Information Systems
  H.2 DATABASE MANAGEMENT
      H.2.4 Systems
          Subjects: Textual databases
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Retrieval models

  J. Computer Applications

General Terms:
Design, Documentation, Management, Performance, Reliability, Theory