Improving text collection selection with coverage and overlap statistics

ABSTRACT

In an environment of distributed text collections, the first step in the information retrieval process is to identify which of all available collections are more relevant to a given query and which should thus be accessed to answer the query. We address the challenge of collection selection when there is full or partial overlap between the available text collections, a scenario which has not been examined previously despite its real-world applications. To that end, we present COSCO, a collection selection approach which uses collection-specific coverage and overlap statistics. We describe our experimental results which show that the presented approach displays the desired behavior of retrieving more new results early on in the collection order, and performs consistently and significantly better than CORI, previously considered to be one of the best collection selection systems.

REFERENCES

• 1: BibFinder: A Computer Science Bibliography Mediator. http://rakaposhi.eas.asu.edu/bibfinder, 2004.
• 2: James P. Callan , Zhihong Lu , W. Bruce Croft, Searching distributed collections with inference networks, Proceedings of the 18th annual international ACM SIGIR conference on Research and development in information retrieval, p.21-28, July 09-13, 1995, Seattle, Washington, United States [doi>10.1145/215206.215328]
• 3: T. Hernandez and S. Kambhampati. Collection selection in the presence of overlapping text collections. Submitted to ACM SIGIR, 2005. http://rakaposhi.eas.asu.edu/cosco.html
• 4: Zaiging Nie , Subbarao Kambhampati, A Frequency-based Approach for Mining Coverage Statistics in Data Integration, Proceedings of the 20th International Conference on Data Engineering, p.387, March 30-April 02, 2004
• 5: Allison L. Powell , James C. French, Comparing the performance of collection selection algorithms, ACM Transactions on Information Systems (TOIS), v.21 n.4, p.412-456, October 2003 [doi>10.1145/944012.944016]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Retrieval models

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Selection process

General Terms:
Management

Keywords:
collection overlap, collection selection, statistics gathering