SDLIP + STARTS = SDARTS a protocol and toolkit for metasearching

ABSTRACT

In this paper we describe how we combined SDLIP and STARTS, two comple mentary protocols for searching over distributed document collections. The resulting protocol, which we call SDARTS, is simple yet expressible enough to enable building sophisticated metasearch engines. SDARTS can be viewed as an instantiation of SDLIP with metasearch-specific elements from STARTS. We also report on our experience building three SDARTS-compliant wrappers: for locally available plain-text document collections, for locally available XML document collections, and for external web-accessible collections. These wrappers were developed to be easily customizable for new collections. Our work was developed as part of Columbia University's Digital Libraries Initiative--Phase 2 (DLI2) project, which involves the departments of Computer Science, Medical Informatics, and Electrical Engineering, the Columbia University libraries, and a large number of industrial partners. The main goal of the project is to provide personalized access to a distributed patient-care digital library.

REFERENCES

• 1: International Standard Maintenance Agency. Z39.50 Maintenance Agency Page. Accessible at http://www.loc.gov/z3950/agency/. ISMA, 2000.
• 2: C. Blake and C. Merz. University of California at Irvine repository of machine learning databases. Accessible at http://kdd.ics.uci.edu/.
• 3: C. M. Bowman, P. B. Danzig, D. R. Hardy, U. Manber, and M. F. Schwartz. Harvest: A scalable, customizable discovery and access system. Technical Report CU-CS-732-94, Department of Computer Science, University of Colorado-Boulder, Aug. 1994.
• 4: Jamie Callan , Margaret Connell , Aiqun Du, Automatic discovery of language models for text databases, Proceedings of the 1999 ACM SIGMOD international conference on Management of data, p.479-490, May 31-June 03, 1999, Philadelphia, Pennsylvania, United States
• 5: James P. Callan , Zhihong Lu , W. Bruce Croft, Searching distributed collections with inference networks, Proceedings of the 18th annual international ACM SIGIR conference on Research and development in information retrieval, p.21-28, July 09-13, 1995, Seattle, Washington, United States [doi>10.1145/215206.215328]
• 6: Jacques Calmet , Sebastian Jekutsch , Joachim Schü, A Generic Query-Translation Framework for a Mediator Architecture, Proceedings of the Thirteenth International Conference on Data Engineering, p.434-443, April 07-11, 1997
• 7: Chen-Chuan K. Chang , Héctor García-Molina, Mind your vocabulary: query mapping across heterogeneous information sources, Proceedings of the 1999 ACM SIGMOD international conference on Management of data, p.335-346, May 31-June 03, 1999, Philadelphia, Pennsylvania, United States
• 8: Kevin Chen-Chuan Chang , Hector Garcia-Molina, Approximate Query Translation Across Heterogeneous Information Sources, Proceedings of the 26th International Conference on Very Large Data Bases, p.566-577, September 10-14, 2000
• 9: E. Christian. Application profile for the government information locator service GILS, Version 2, Aug. 1997. Accessible at http://www.usgs.gov/gils/prof v2.html.
• 10: Nick Craswell , Peter Bailey , David Hawking, Server selection on the World Wide Web, Proceedings of the fifth ACM conference on Digital libraries, p.37-46, June 02-07, 2000, San Antonio, Texas, United States [doi>10.1145/336597.336628]
• 11: Luis Gravano , Chen-Chuan K. Chang , Héctor García-Molina , Andreas Paepcke, STARTS: Stanford proposal for Internet meta-searching, Proceedings of the 1997 ACM SIGMOD international conference on Management of data, p.207-218, May 11-15, 1997, Tucson, Arizona, United States
• 12: Luis Gravano , Héctor García-Molina , Anthony Tomasic, GlOSS: text-source discovery over the Internet, ACM Transactions on Database Systems (TODS), v.24 n.2, p.229-264, June 1999 [doi>10.1145/320248.320252]
• 13: David Hawking , Paul Thistlewaite, Methods for information server selection, ACM Transactions on Information Systems (TOIS), v.17 n.1, p.40-76, Jan. 1999 [doi>10.1145/297117.297123]
• 14: HTML Tidy. Accessible at http://www.w3.org/People/Raggett/tidy/, 2000.
• 15: Panagiotis G. Ipeirotis , Luis Gravano , Mehran Sahami, Probe, count, and classify: categorizing hidden web databases, Proceedings of the 2001 ACM SIGMOD international conference on Management of data, p.67-78, May 21-24, 2001, Santa Barbara, California, United States
• 16: Hartmut Liefke , Dan Suciu, XMill: an efficient compressor for XML data, Proceedings of the 2000 ACM SIGMOD international conference on Management of data, p.153-164, May 15-18, 2000, Dallas, Texas, United States
• 17: The Lucene Search Engine. Accessible at http://www.lucene.com/, 2000.
• 18: Weiyi Meng , King-Lup Liu , Clement T. Yu , Xiaodong Wang , Yuhsi Chang , Naphtali Rishe, Determining Text Databases to Search in the Internet, Proceedings of the 24rd International Conference on Very Large Data Bases, p.14-25, August 24-27, 1998
• 19: Open Archives Initiative. Accessible at http://www.openarchives.org/, 2000.
• 20: A. Paepcke, R. Brandriff, G. Janee, R. Larson, B. Ludaescher, S. Melnik, and S. Raghavan. Search middleware and the Simple Digital Library Interoperability Protocol. D-Lib Magazine, 6(3), 2000.
• 21: Atsushi Sugiura , Oren Etzioni, Query routing for Web search engines: architectures and experiments, Proceedings of the 9th international World Wide Web conference on Computer networks : the international journal of computer and telecommunications netowrking, p.417-429, June 2000, Amsterdam, The Netherlands
• 22: E. M. Voorhees, N. K. Gupta, and B. Johnson-Laird. The collection fusion problem. In Overview of the Third Text REtrieval Conference (TREC-3), pages 95-104. Department of Commerce, National Institute of Standards and Technology, Mar. 1995.
• 23: S. Weibel, J. Godby, E. Miller, and R. Daniel Jr. OCLC/NCSA metadata workshop report, 1995. Accessible at http://www.oclc.org:5047/oclc/- research/publications/weibel/metadata/- dublin core report.html.
• 24: Jinxi Xu , Jamie Callan, Effective retrieval with distributed collections, Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, p.112-120, August 24-28, 1998, Melbourne, Australia [doi>10.1145/290941.290974]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL

Additional Classification:
  C. Computer Systems Organization
  C.2 COMPUTER-COMMUNICATION NETWORKS

  H. Information Systems
  H.2 DATABASE MANAGEMENT
      H.2.4 Systems
          Subjects: Distributed databases
          Nouns: Database Manager
  H.3 INFORMATION STORAGE AND RETRIEVAL

General Terms:
Design, Documentation, Human Factors, Management, Performance, Standardization, Theory