The infocious web search engine: improving web searching through linguistic analysis

ABSTRACT

In this paper we present the Infocious Web search engine [23]. Our goal in creating Infocious is to improve the way people find information on the Web by resolving ambiguities present in natural language text. This is achieved by performing linguistic analysis on the content of the Web pages we index, which is a departure from existing Web search engines that return results mainly based on keyword matching. This additional step of linguistic processing gives Infocious two main advantages. First, Infocious gains a deeper understanding of the content of Web pages so it can better match users' queries with indexed documents and therefore can improve relevancy of the returned results. Second, based on its linguistic processing, Infocious can organize and present the results to the user in more intuitive ways. In this paper we present the linguistic processing technologies that we incorporated in Infocious and how they are applied in helping users find information on the Web more efficiently. We discuss the various components in the architecture of Infocious and how each of them benefits from the added linguistic processing. Finally, we experimentally evaluate the performance of a component which leverages linguistic information in order to categorize Web pages.

REFERENCES

• 1: Eugene Agichtein , Steve Lawrence , Luis Gravano, Learning to find answers to questions on the Web, ACM Transactions on Internet Technology (TOIT), v.4 n.2, p.129-162, May 2004 [doi>10.1145/990301.990303]
• 2: Altavista Inc. http://www.altavista.com.
• 3: Ask Jeeves Inc. http://www.ask.com.
• 4: Autonomy Inc. http://www.autonomy.com.
• 5: Ricardo A. Baeza-Yates , Berthier Ribeiro-Neto, Modern Information Retrieval, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1999
• 6: Brainboost. http://www.brainboost.com.
• 7: Eric Brill, Some advances in transformation-based part of speech tagging, Proceedings of the twelfth national conference on Artificial intelligence (vol. 1), p.722-727, October 1994, Seattle, Washington, United States
• 8: Gerald Che-Shun Chao , Michael G. Dyer, A probabilistic, integrative approach for improved natural language disambiguation, 2003
• 9: Gerald Chao , Michael G. Dyer, Maximum entropy models for word sense disambiguation, Proceedings of the 19th international conference on Computational linguistics, p.1-7, August 24-September 01, 2002, Taipei, Taiwan [doi>10.3115/1072228.1072396]
• 10: C. Chekuri, M. Goldwasser, P. Raghavan, and E. Upfal. Web search using automatic classification. In Proceedings of WWW-96, 6th International Conference on the World Wide Web, San Jose, US, 1996.
• 11: Junghoo Cho , Hector Garcia-Molina, Crawling the web: discovery and maintenance of large-scale web data, 2002
• 12: Junghoo Cho , Hector Garcia-Molina, The Evolution of the Web and Implications for an Incremental Crawler, Proceedings of the 26th International Conference on Very Large Data Bases, p.200-209, September 10-14, 2000
• 13: Junghoo Cho , Hector Garcia-Molina, Synchronizing a database to improve freshness, Proceedings of the 2000 ACM SIGMOD international conference on Management of data, p.117-128, May 15-18, 2000, Dallas, Texas, United States
• 14: J. Cho and A. Ntoulas. Effective change detection using sampling. In Proceedings of the Twenty-eighth International Conference on Very Large Databases (VLDB), August 2002.
• 15: Nello Cristianini , John Shawe-Taylor, An introduction to support Vector Machines: and other kernel-based learning methods, Cambridge University Press, New York, NY, 1999
• 16: S. Deerwester, S. T. Dumais, G. W. Furnas, T. K. Landauer, and R. Harshman. Indexing by latent semantic analysis. Journal of the American Society for Information Science, 41(6):391--407, September 1990.
• 17: The open directory project. http://www.dmoz.org.
• 18: R. O. Duda and P. E. Hart. Pattern Classification and Scene Analysis. Wiley, New York, 1973.
• 19: S. T. Dumais. Latent semantic indexing (LSI) and TREC-2. In The Second Text Retrieval Conference (TREC-2), 1994.
• 20: Excite Inc. http://www.excite.com.
• 21: Google Incorporated. http://www.google.com.
• 22: Chien-Chung Huang , Shui-Lung Chuang , Lee-Feng Chien, Liveclassifier: creating hierarchical text classifiers through web corpora, Proceedings of the 13th international conference on World Wide Web, May 17-20, 2004, New York, NY, USA [doi>10.1145/988672.988698]
• 23: Infocious Incorporated. http://www.infocious.com.
• 24: Inquira Inc. http://www.inquira.com.
• 25: Inxight Inc. http://www.inxight.com.
• 26: iPhrase Inc. http://www.iphrase.com.
• 27: Daniel Jurafsky , James H. Martin, Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition, Prentice Hall PTR, Upper Saddle River, NJ, 2000
• 28: B. Katz, J. Lin, D. Loreto, W. Hildebrandt, M. Bilotti, S. Felshin, A. Fernandes, G. Marton, and F. Mora. Integrating web-based and corpus-based techniques for question answering, November 2003.
• 29: C. Li, J.-R. Wen, and H. Li. Text classification using stochastic keyword generation. In Twentieth International Conference on Machine Learning (ICML), pages 464--471, 2003.
• 30: Lycos Inc. http://www.lycos.com.
• 31: Christopher D. Manning , Hinrich Schütze, Foundations of statistical natural language processing, MIT Press, Cambridge, MA, 1999
• 32: O. A. McBryan. GENVL and WWWW: Tools for taming the web. In First International Conference on the World Wide Web, CERN, Geneva, Switzerland, May 1994.
• 33: R. Mihalcea. Bootstrapping large sense tagged corpora. In Proceedings of the 3rd International Conference on Language Resources and Evaluations (LREC), Las Palmas, Spain, May 2002.
• 34: MSNSearch. http://www.msnsearch.com.
• 35: Kamal Nigam , Andrew Kachites McCallum , Sebastian Thrun , Tom Mitchell, Text Classification from Labeled and Unlabeled Documents using EM, Machine Learning, v.39 n.2-3, p.103-134, May-June 2000 [doi>10.1023/A:1007692713085]
• 36: A. Ntoulas, P. Zerfos, and J. Cho. Downloading hidden web content. Technical report, UCLA, 2004. Available at http://oak.cs.ucla.edu/~ntoulas/pubs/ntoulas_hidden_web_extended.pdf.
• 37: L. Page, S. Brin, R. Motwani, and T. Winograd. The pagerank citation ranking: Bringing order to the web. Technical Report 1999-66, Database Group, Computer Science Department, Stanford University, November 1999. http://dbpubs.stanford.edu/pub/1999-66.
• 38: A. Ratnaparkhi. A maximum entropy model for part-of-speech tagging. In Proceedings of the First Conference on Empirical Methods in Natural Language Processing, pages 133--142, 1996.
• 39: Gerard Salton , Michael J. McGill, Introduction to Modern Information Retrieval, McGraw-Hill, Inc., New York, NY, 1986
• 40: START natural language question answering system. http://www.ai.mit.edu/projects/infolab/.
• 41: Teoma. http://www.teoma.com.
• 42: Scott M. Thede , Mary P. Harper, A second-order Hidden Markov Model for part-of-speech tagging, Proceedings of the 37th annual meeting of the Association for Computational Linguistics on Computational Linguistics, p.175-182, June 20-26, 1999, College Park, Maryland [doi>10.3115/1034678.1034712]
• 43: Erik F. Tjong Kim Sang , Sabine Buchholz, Introduction to the CoNLL-2000 shared task: chunking, Proceedings of the 2nd workshop on Learning language in logic and the 4th conference on Computational natural language learning, September 13-14, 2000, Lisbon, Portugal [doi>10.3115/1117601.1117631]
• 44: A. J. Viterbi. Error bounds for convolutional codes and an asymtotically optimum decoding algorithm. IEEE Transactions on Information Theory, IT-13:260--267, 1967.
• 45: Ian H. Witten , Alistair Moffat , Timothy C. Bell, Managing gigabytes (2nd ed.): compressing and indexing documents and images, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1999
• 46: Yahoo! Inc. http://www.yahoo.com.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.1 Content Analysis and Indexing

Additional Classification:
  C. Computer Systems Organization
  C.3 SPECIAL-PURPOSE AND APPLICATION-BASED SYSTEMS

  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
      H.3.7 Digital Libraries

Keywords:
concept extraction, crawling, indexing, information retrieval, language analysis, linguistic analysis of web text, natural language processing, part-of-speech tagging, phrase identification, web search engine, web searching, word sense disambiguation