PowerDB-IR: information retrieval on top of a database cluster

ABSTRACT

Our current concern is a scalable infrastructure for information retrieval (IR) with up-to-date retrieval results in the presence of frequent, continuous updates. Timely processing of updates is important with novel application domains, e.g., e-commerce. We want to use off-the-self hardware and software as much as possible. These issues are challenging, given the additional requirement that the resulting system must scale well. We have built PowerDB-IR, a system that has the characteristics sought. This paper describes its design, implementation, and evaluation. PowerDB-IR is a coordination layer for a database cluster. The rationale behind a database cluster is to 'scale-out', i.e., to add further cluster nodes, whenever necessary for better performance. We build on IR-to-database mappings and service decomposition to support high-level parallelism. We follow a three-tier architecture with the database cluster as the bottom layer for storage management. The middle tier provides IR-specific processing and update services. PowerDB-IR has the following features: It allows to insert and retrieve documents concurrently, and it ensures freshness with almost no overhead. Alternative physical data organization schemes provide adequate performance for different workloads. Query processing techniques for the different data organizations efficiently integrate the ranked retrieval results from the cluster nodes. We have run extensive experiments with our prototype using commercial database systems and middleware software products. The main result is that PowerDB-IR shows surprisingly ideal scalability and low response times.

REFERENCES

• 1: Daniel Andresen , Tao Yang , Oscar H. Ibarra, Toward a scalable distributed WWW server on workstation clusters, Journal of Parallel and Distributed Computing, v.42 n.1, p.91-100, April 10, 1997 [doi>10.1006/jpdc.1997.1305]
• 2: Daniel Barbará , Sharad Mehrotra , Padmavathi Vallabhaneni, The Gold Text Indexing Engine, Proceedings of the Twelfth International Conference on Data Engineering, p.172-179, February 26-March 01, 1996
• 3: Klemens Böhm , Karl Aberer , Erich J. Neuhold , Xiaoya Yang, Structured document storage and refined declarative and navigational access mechanisms in HyperStorM, The VLDB Journal — The International Journal on Very Large Data Bases, v.6 n.4, p.296-311, November 1997 [doi>10.1007/s007780050047]
• 4: Eric W. Brown , James P. Callan , W. Bruce Croft, Fast Incremental Indexing for Full-Text Information Retrieval, Proceedings of the 20th International Conference on Very Large Data Bases, p.192-202, September 12-15, 1994
• 5: George Copeland , William Alexander , Ellen Boughter , Tom Keller, Data placement in Bubba, Proceedings of the 1988 ACM SIGMOD international conference on Management of data, p.99-108, June 01-03, 1988, Chicago, Illinois, United States
• 6: S. DeFazio. Overview of the Full-Text Document Retrieval Benchmark (In: The Benchmark Handbook-Jim Gray (ed.)), pages 435487. Morgan Kaufmann, 199 1.
• 7: Armando Fox , Steven D. Gribble , Yatin Chawathe , Eric A. Brewer , Paul Gauthier, Cluster-based scalable network services, Proceedings of the sixteenth ACM symposium on Operating systems principles, p.78-91, October 05-08, 1997, Saint Malo, France
• 8: O. Frieder, A. Chowdhury, D. Grossman, and M. McCabe. On the Integration of Structured Data and Text: A Review of the SIRE Architecture. In Proceedings of the First DELOS Network of Excellence Workshop on Information Seeking, Searching and Querying in Digital Libraries, Zurich, Switzerland, 2000, pages 53-58. ERCIM, 2000.
• 9: Torsten Grabs , Klemens Böhm , Hans-Jörg Schek, High-level Parallelism in a Database Cluster: A Feasibility Study Using Document Services, Proceedings of the 17th International Conference on Data Engineering, p.121-130, April 02-06, 2001
• 10: T. Grabs, K. Bohm, and H.-J. Schek. PowerDB-IR- Information Retrieval on Top of a Database Cluster. Technical report, Database Research Group, ETH Zurich, 200 1. Available at: http://www.dbs.ethz.ch/~grabs/papers/cikm2001 long.pdf.
• 11: Torsten Grabs , Klemens Böhm , Hans-Jörg Schek, Scalable Distributed Query and Update Service Implementations for XML Document Elements, Eleventh International Workshop on Research Issues in Data Engineering on Document Management for Data Intensive Business and Scientific Applications, p.35-42, April 01-02, 2001
• 12: J. Gray. How High is High Performance Transaction Processing. In High Performance Transaction Systems Workshop, Asilomar. USA, 1999. Available at: http://research.microsoftt.com/~gray/hpts99/talks/-GrayJim. ppt.
• 13: David A. Grossman , Ophir Frieder , David O. Holmes , David C. Roberts, Integrating structured data and text: a relational approach, Journal of the American Society for Information Science, v.48 n.2, p.122-132, Feb. 1997 [doi>10.1002/(SICI)1097-4571(199702)48:2<122::AID-ASI3>3.0.CO;2-#]
• 14: Inktomi Corp. The Inktomi Technology behind HotBot. Technical report, Inktomi Corp., 1996.
• 15: Steven Kirsch, Infoseek's experiences searching the internet, ACM SIGIR Forum, v.32 n.2, p.3-7, Sept. 1998 [doi>10.1145/305110.305112]
• 16: Microsoft Corp. Building High-Performance Databases Using Microsoft SQL Server 2000 Federated Database Servers. Technical report, Microsoft Corp., 2000.
• 17: Hans-Jörg Schek , Peter Pistor, Data Structures for an Integrated Data Base Management and Information Retrieval System, Proceedings of the 8th International Conference on Very Large Data Bases, p.197-207, September 08-10, 1982
• 18: Anthony Tomasic , Héctor García-Molina , Kurt Shoens, Incremental updates of inverted lists for text document retrieval, Proceedings of the 1994 ACM SIGMOD international conference on Management of data, p.289-300, May 24-27, 1994, Minneapolis, Minnesota, United States

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.2 DATABASE MANAGEMENT
      H.2.7 Database Administration
          Subjects: Data warehouse and repository

Additional Classification:
  H. Information Systems
  H.2 DATABASE MANAGEMENT
      H.2.4 Systems
          Subjects: Query processing
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Retrieval models

General Terms:
Algorithms, Design, Performance