Document relevance assessment via term distribution analysis using fourier series expansion

ABSTRACT

In addition to the frequency of terms in a document collection, the distribution of terms plays an important role in determining the relevance of documents for a given search query. In this paper, term distribution analysis using Fourier series expansion as a novel approach for calculating an abstract representation of term positions in a document corpus is introduced. Based on this approach, two methods for improving the evaluation of document relevance are proposed: (a) a function-based ranking optimization representing a user defined document region, and (b) a query expansion technique based on overlapping the term distributions in the top-ranked documents. Experimental results demonstrate the effectiveness of the proposed approach in providing new possibilities for optimizing the retrieval process.

REFERENCES

• 1: Gianni Amati , Cornelis Joost Van Rijsbergen, Probabilistic models of information retrieval based on measuring the divergence from randomness, ACM Transactions on Information Systems (TOIS), v.20 n.4, p.357-389, October 2002 [doi>10.1145/582415.582416]
• 2: R. Attar and A. Fraenkel. Experiments in local metrical feedback in full-text retrieval systems. Information Processing and Management, 17(3):115--126, 1981.
• 3: R. Attar , A. S. Fraenkel, Local Feedback in Full-Text Retrieval Systems, Journal of the ACM (JACM), v.24 n.3, p.397-417, July 1977 [doi>10.1145/322017.322021]
• 4: Ricardo A. Baeza-Yates , Berthier Ribeiro-Neto, Modern Information Retrieval, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1999
• 5: Michael W. Berry , Susan T. Dumais , Gavin W. O'Brien, Using linear algebra for intelligent information retrieval, SIAM Review, v.37 n.4, p.573-595, Dec. 1995 [doi>10.1137/1037127]
• 6: Bodo Billerbeck , Falk Scholer , Hugh E. Williams , Justin Zobel, Query expansion using associated queries, Proceedings of the twelfth international conference on Information and knowledge management, November 03-08, 2003, New Orleans, LA, USA [doi>10.1145/956863.956866]
• 7: Chris Buckley , Gerard Salton , James Allan, The effect of adding relevance information in a relevance feedback environment, Proceedings of the 17th annual international ACM SIGIR conference on Research and development in information retrieval, p.292-300, July 03-06, 1994, Dublin, Ireland
• 8: C. Buckley, G. Salton, J. Allan, and A. Singhal. Automatic query expansion using smart: TREC-3. In Overview of the 3rd Text Retrieval Conference, pages 69--80. NIST Special Publication, 1995.
• 9: Deng Cai , Shipeng Yu , Ji-Rong Wen , Wei-Ying Ma, Block-based web search, Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval, July 25-29, 2004, Sheffield, United Kingdom [doi>10.1145/1008992.1009070]
• 10: Thomas M. Cover , Joy A. Thomas, Elements of information theory, Wiley-Interscience, New York, NY, 1991
• 11: E. Efthimiadis. Interactive query expansion and relevance feedback for document retrieval systems. PhD thesis, City University, London UK, 1992.
• 12: E. Efthimiadis. Query expansion. Annual Review of Information Science and Technology (ARIST), (2):121--187, 1996.
• 13: E. Efthimiadis and P. Biron. Ucla-okapi at TREC-2: Query expansion experiments. In Proceedings of the 2nd Text Retrieval Conference (TREC-2), pages 279--290. NIST Special Publication 500-215, 1994.
• 14: Weiguo Fan , Ming Luo , Li Wang , Wensi Xi , Edward A. Fox, Tuning before feedback: combining ranking discovery and blind feedback for robust retrieval, Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval, July 25-29, 2004, Sheffield, United Kingdom [doi>10.1145/1008992.1009018]
• 15: Placing search in context: the concept revisited, ACM Transactions on Information Systems (TOIS), v.20 n.1, p.116-131, January 2002 [doi>10.1145/503104.503110]
• 16: Andre Folkers , Hanan Samet, Content-Based Image Retrieval Using Fourier Descriptors on a Logo Database, Proceedings of the 16 th International Conference on Pattern Recognition (ICPR'02) Volume 3, p.30521, August 11-15, 2002
• 17: Marti A. Hearst , Jan O. Pedersen, Reexamining the cluster hypothesis: scatter/gather on retrieval results, Proceedings of the 19th annual international ACM SIGIR conference on Research and development in information retrieval, p.76-84, August 18-22, 1996, Zurich, Switzerland [doi>10.1145/243199.243216]
• 18: X. Huang and Y. Huang. Using contextual information to improve retrieval performance. In Proceedings of 2005 IEEE International Conference on Granular Computing, pages 474--481, Beijing, China, July 2005.
• 19: U. N. Jonathan Foote, Matthew D. Cooper. Audio retrieval by rhythmic similarity. In 3rd International Conference on Music Information Retrieval, 2002.
• 20: Marcin Kaszkiel , Justin Zobel, Passage retrieval revisited, Proceedings of the 20th annual international ACM SIGIR conference on Research and development in information retrieval, p.178-185, July 27-31, 1997, Philadelphia, Pennsylvania, United States
• 21: Slava M. Katz, Distribution of content words and phrases in text and language modelling, Natural Language Engineering, v.2 n.1, p.15-59, March 1996 [doi>10.1017/S1351324996001246]
• 22: Laurence A. F. Park , Kotagiri Ramamohanarao , Marimuthu Palaniswami, Fourier Domain Scoring: A Novel Document Ranking Method, IEEE Transactions on Knowledge and Data Engineering, v.16 n.5, p.529-539, May 2004 [doi>10.1109/TKDE.2004.1277815]
• 23: S. Lawrence. Context in web search. IEEE Data Engineering Bulletin, 23(3):25--32, 2000.
• 24: J. Li, M. Guo, and S. Tian. A new approach to query expansion. In Machine Learning and Cybernetics, pages 2302--2306, August 2005.
• 25: David R. H. Miller , Tim Leek , Richard M. Schwartz, A hidden Markov model information retrieval system, Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval, p.214-221, August 15-19, 1999, Berkeley, California, United States [doi>10.1145/312624.312680]
• 26: David S. Moore, The Basic Practice of Statistics with Cdrom, W. H. Freeman & Co., New York, NY, 1999
• 27: I. Ounis, G. Amati, V. Plachouras, B. He, C. Macdonald, and C. Lioma. Terrier: A high performance and scalable information retrieval platform. In Proceedings of ACM SIGIR'06 Workshop on Open Source Information Retrieval (OSIR 2006), 2006.
• 28: M. A. Razek, C. Frasson, and M. Kaltenbach. Context-based information agent for supporting intelligent distance learning environment. In Proc. of the Twelfth International World Wide Web Conference, WWW03, page 968, Budapest, Hungary, 2003. Springer-Verlag.
• 29: S. Robertson and K. S. Jones. Relevance weighting of search terms. American Society for Information Sciences, 27(3):129--146, 1976.
• 30: J. Rocchio. Relevance feedback in information retrieval. In G. Salton, editor, The SMART Retrieval System: Experiments in Automatic Document Processing, pages 313--323, Englewood Cliffs, NJ, 1971. Prentice-Hall.
• 31: Xuehua Shen , Cheng Xiang Zhai, Exploiting query history for document ranking in interactive information retrieval, Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval, July 28-August 01, 2003, Toronto, Canada [doi>10.1145/860435.860509]
• 32: M. R. Spiegel. Schaum's Outline of theory and problems of Fourier analysis. McGraw Hill, New York, 1 edition, 1974.
• 33: Renxu Sun , Chai-Huat Ong , Tat-Seng Chua, Mining dependency relations for query expansion in passage retrieval, Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, August 06-11, 2006, Seattle, Washington, USA [doi>10.1145/1148170.1148237]
• 34: Tao Tao , ChengXiang Zhai, An exploration of proximity measures in information retrieval, Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, July 23-27, 2007, Amsterdam, The Netherlands [doi>10.1145/1277741.1277794]
• 35: Jinxi Xu , W. Bruce Croft, Improving the effectiveness of information retrieval with local context analysis, ACM Transactions on Information Systems (TOIS), v.18 n.1, p.79-112, Jan. 2000 [doi>10.1145/333135.333138]
• 36: Jinxi Xu , Ralph Weischedel, Cross-lingual information retrieval using hidden Markov models, Proceedings of the 2000 Joint SIGDAT conference on Empirical methods in natural language processing and very large corpora: held in conjunction with the 38th Annual Meeting of the Association for Computational Linguistics, p.95-103, October 07-08, 2000, Hong Kong [doi>10.3115/1117794.1117806]
• 37: Shipeng Yu , Deng Cai , Ji-Rong Wen , Wei-Ying Ma, Improving pseudo-relevance feedback in web information retrieval using web page segmentation, Proceedings of the 12th international conference on World Wide Web, May 20-24, 2003, Budapest, Hungary [doi>10.1145/775152.775155]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Retrieval models

General Terms:
Algorithms, Experimentation

Keywords:
fourier series, query expansion, ranked retrieval, term distribution