Information retrieval from digital libraries in SQL

ABSTRACT

Information retrieval techniques have been traditionally exploited outside of relational database systems, due to storage overhead, the complexity of programming them inside the database system, and their slow performance in SQL implementations. This project supports the idea that searching and querying digital libraries with information retrieval models in relational database systems can be performed with optimized SQL queries and User-Defined Functions. In our research, we propose several techniques divided into two phases: storing and retrieving. The storing phase includes executing document pre-processing, stop-word removal and term extraction, and the retrieval phase is implemented with three fundamental IR models: the popular Vector Space Model, the Okapi Probabilistic Model, and the Dirichlet Prior Language Model. We conduct experiments using article abstracts from the DBLP bibliography and the ACM Digital Library. We evaluate several query optimizations, compare the on-demand and the static weighting approaches, and we study the performance with conjunctive and disjunctive queries with the three ranking models. Our prototype proved to have linear scalability and a satisfactory performance with medium-sized document collections. Our implementation of the Vector Space Model is competitive with the two other models.

REFERENCES

• 1: J. Driscoll D.A. Grossman. Structuring text within a relational system. DEXA '92, September 1992.
• 2: Hui Fang , Tao Tao , ChengXiang Zhai, A formal study of information retrieval heuristics, Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval, July 25-29, 2004, Sheffield, United Kingdom [doi>10.1145/1008992.1009004]
• 3: K. Goda , T. Tamura , M. Kitsuregawa , A. Chowdhury , O. Frieder, Query optimization for vector space problems, Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval, p.416-417, September 2001, New Orleans, Louisiana, United States [doi>10.1145/383952.384064]
• 4: Torsten Grabs , Klemens Böhm , Hans-Jörg Schek, PowerDB-IR: information retrieval on top of a database cluster, Proceedings of the tenth international conference on Information and knowledge management, October 05-10, 2001, Atlanta, Georgia, USA [doi>10.1145/502585.502655]
• 5: Torsten Grabs , Klemens Böhm , Hans-Jörg Schek, PowerDB-IR – Scalable Information Retrieval and Storage with a Cluster of Databases, Knowledge and Information Systems, v.6 n.4, p.465-505, July 2004 [doi>10.1007/s10115-003-0120-y]
• 6: D.A. Grossman, D.O. Holmes, and O. Frieder. A parallel DBMS approach to ir in TREC-3. In Text REtrieval Conference, 1994.
• 7: David A. Grossman , Ophir Frieder, Information Retrieval: Algorithms and Heuristics (The Kluwer International Series on Information Retrieval), Springer-Verlag New York, Inc., Secaucus, NJ, 2004
• 8: David Holmes, SQL text parsing for information retrieval, Proceedings of the twelfth international conference on Information and knowledge management, November 03-08, 2003, New Orleans, LA, USA [doi>10.1145/956863.956956]
• 9: M. Porter K. Lubell. Porter algorithm T-SQL. Url: http://tartarus.org/ martin/PorterStemmer/tsql.txt, May 2006.
• 10: Fang Liu , Clement Yu , Weiyi Meng , Abdur Chowdhury, Effective keyword search in relational databases, Proceedings of the 2006 ACM SIGMOD international conference on Management of data, June 27-29, 2006, Chicago, IL, USA [doi>10.1145/1142473.1142536]
• 11: D.A. Grossman O. Frieder M.C. McCabe, D. Holmes. Parallel platform-independent implementation of information retrieval algorithms. PDPTA'00, 2000.
• 12: A.K. McCallum. Bow: A toolkit for statistical language modeling, text retrieval, classification and clustering. http://www.cs.cmu.edu/ mccallum/bow, 1996.
• 13: A.K. McCallum. Mallet: A machine learning for language toolkit. http://mallet.cs.umass.edu, 2002.
• 14: Mauricio Marin , Carlos Gomez, Load balancing distributed inverted files, Proceedings of the 9th annual ACM international workshop on Web information and data management, November 09-09, 2007, Lisbon, Portugal [doi>10.1145/1316902.1316912]
• 15: Carlos Ordonez, Optimizing recursive queries in SQL, Proceedings of the 2005 ACM SIGMOD international conference on Management of data, June 14-16, 2005, Baltimore, Maryland [doi>10.1145/1066157.1066260]
• 16: Carlos Ordonez , Javier García-García, Referential integrity quality metrics, Decision Support Systems, v.44 n.2, p.495-508, January, 2008 [doi>10.1016/j.dss.2007.06.004]
• 17: Chris D. Paice, Another stemmer, ACM SIGIR Forum, v.24 n.3, p.56-61, Fall 1990 [doi>10.1145/101306.101310]
• 18: G. Salton , A. Wong , C. S. Yang, A vector space model for automatic indexing, Communications of the ACM, v.18 n.11, p.613-620, Nov. 1975 [doi>10.1145/361219.361220]
• 19: Thi Truong Avrahami , Lawrence Yau , Luo Si , Jamie Callan, The FedLemur project: Federated search in the real world, Journal of the American Society for Information Science and Technology, v.57 n.3, p.347-358, February 2006 [doi>10.1002/asi.v57:3]
• 20: G. Weikum. DB&IR: Both sides now. pages 25--30, 2007.
• 21: Justin Zobel , Alistair Moffat, Inverted files for text search engines, ACM Computing Surveys (CSUR), v.38 n.2, p.6-es, 2006 [doi>10.1145/1132956.1132959]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval

Additional Classification:
  H. Information Systems
  H.2 DATABASE MANAGEMENT
      H.2.8 Database applications

General Terms:
Algorithms, Languages

Keywords:
SQL, documents, query optimization, ranking