Shallow NLP techniques for internet search

ABSTRACT

Information Retrieval (IR) is a major component in many of our daily activities, with perhaps its most prominent role manifested in search engines. Today's most advanced engines use the keyword-based ("bag of words") paradigm, which concedes some inherent disadvantages. We believe that natural language (NL) is a more user-oriented, context-preservative and intuitive mechanism for web search.In this paper, we explore shallow NLP techniques to support a range of NL queries over an existing keyword-based engine. We present JASE, a web application enveloping the Google search engine, which performs web searches by decomposing input NL queries and generating new queries that are more suitable for the search engine. By using some of Google's syntactic operators and filters, it creates "clever" queries to improve precision.A preliminary evaluation was conducted to test JASE's accuracy, and results have been encouraging. We conclude that the NL model has potential to not only rival the keyword-based paradigm, but substantially surpass it.

REFERENCES

• 1: Steven Abney , Michael Collins , Amit Singhal, Answer extraction, Proceedings of the sixth conference on Applied natural language processing, p.296-301, April 29-May 04, 2000, Seattle, Washington [doi>10.3115/974147.974188]
• 2: Eugene Agichtein , Steve Lawrence , Luis Gravano, Learning search engine specific query transformations for question answering, Proceedings of the 10th international conference on World Wide Web, p.169-178, May 01-05, 2001, Hong Kong, Hong Kong [doi>10.1145/371920.371976]
• 3: Eric Brill, A simple rule-based part of speech tagger, Proceedings of the third conference on Applied natural language processing, March 31-April 03, 1992, Trento, Italy [doi>10.3115/974499.974526]
• 4: Eric Brill, Transformation-based error-driven learning and natural language processing: a case study in part-of-speech tagging, Computational Linguistics, v.21 n.4, p.543-565, December 1995
• 5: Charniak, E. (1994), Statistical language learning, in 'Language and Computers 12', The MIT Press.
• 6: Charniak, E. (1997), 'Statistical techniques for natural language parsing', AI Magazine 18(4), 33-44.
• 7: Michael John Collins, A new statistical parser based on bigram lexical dependencies, Proceedings of the 34th annual meeting on Association for Computational Linguistics, p.184-191, June 24-27, 1996, Santa Cruz, California [doi>10.3115/981863.981888]
• 8: Sharon Flank, A layered approach to NLP-based information retrieval, Proceedings of the 17th international conference on Computational linguistics, August 10-14, 1998, Montreal, Quebec, Canada
• 9: Eibe Frank , Gordon W. Paynter , Ian H. Witten , Carl Gutwin , Craig G. Nevill-Manning, Domain-Specific Keyphrase Extraction, Proceedings of the Sixteenth International Joint Conference on Artificial Intelligence, p.668-673, July 31-August 06, 1999
• 10: Jones, K. S. (1972), 'A statistical interpretation of term specificity and its application to retrieval', Journal of Documentation 28(1), 11-21.
• 11: Boris Katz, Text processing with the START natural language system, Text, ConText, and HyperText: writing with and for the computer, MIT Press, Cambridge, MA, 1991
• 12: Boris Katz , Jimmy Lin, REXTOR: a system for generating relations from natural language, Proceedings of the ACL-2000 workshop on Recent advances in natural language processing and information retrieval: held in conjunction with the 38th Annual Meeting of the Association for Computational Linguistics, October 08-08, 2000, Hong Kong [doi>10.3115/1117755.1117764]
• 13: Julian Kupiec, MURAX: a robust linguistic approach for question answering using an on-line encyclopedia, Proceedings of the 16th annual international ACM SIGIR conference on Research and development in information retrieval, p.181-190, June 27-July 01, 1993, Pittsburgh, Pennsylvania, United States [doi>10.1145/160688.160717]
• 14: Cody C. T. Kwok , Oren Etzioni , Daniel S. Weld, Scaling question answering to the Web, Proceedings of the 10th international conference on World Wide Web, p.150-161, May 01-05, 2001, Hong Kong, Hong Kong [doi>10.1145/371920.371973]
• 15: Luhn, H. P. (1957), 'A statistical approach to mechanized encoding and searching of literary information', IBM Journal of Research and Development, 4(4), 600-605.
• 16: Munoz, A. (1996), Compound key word generation from document databases using a hierarchical clustering ART model. IDA, Amsterdam.
• 17: Page, L., Brin, S., Motwani, R. & Winograd, T. (1998), 'The pagerank citation ranking: Bringing order to the web', Stanford Digital Library Technologies Project.
• 18: Porter, M. (1980), An algorithm for suffix stripping, in 'Program', Vol. 14, pp. 130-137.
• 19: John Prager , Eric Brown , Anni Coden , Dragomir Radev, Question-answering by predictive annotation, Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, p.184-191, July 24-28, 2000, Athens, Greece [doi>10.1145/345508.345574]
• 20: Deepak Ravichandran , Eduard Hovy, Learning surface text patterns for a Question Answering system, Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, July 07-12, 2002, Philadelphia, Pennsylvania [doi>10.3115/1073083.1073092]
• 21: Gerard Salton, Automatic text processing: the transformation, analysis, and retrieval of information by computer, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1989
• 22: Gerard Salton , Christopher Buckley, Term-weighting approaches in automatic text retrieval, Information Processing and Management: an International Journal, v.24 n.5, p.513-523, 1988 [doi>10.1016/0306-4573(88)90021-0]
• 23: Turney, P. (1999), 'Learning to extract keyphrases from text', Technical Report ERB-1057, National Research Council, Institute for Information Technology.
• 24: Peter D. Turney, Learning Algorithms for Keyphrase Extraction, Information Retrieval, v.2 n.4, p.303-336, May 2000 [doi>10.1023/A:1009976227802]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Natural language

  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.1 Applications and Expert Systems
          Subjects: Natural language interfaces
      I.2.7 Natural Language Processing

General Terms:
Algorithms, Design, Theory

Keywords:
google, information retrieval, natural language, processing