Learning to rank for quantity consensus queries

ABSTRACT

Web search is increasingly exploiting named entities like persons, places, businesses, addresses and dates. Entity ranking is also of current interest at INEX and TREC. Numerical quantities are an important class of entities, especially in queries about prices and features related to products, services and travel. We introduce Quantity Consensus Queries (QCQs), where each answer is a tight quantity interval distilled from evidence of relevance in thousands of snippets. Entity search and factoid question answering have benefited from aggregating evidence from multiple promising snippets, but these do not readily apply to quantities. Here we propose two new algorithms that learn to aggregate information from multiple snippets. We show that typical signals used in entity ranking, like rarity of query words and their lexical proximity to candidate quantities, are very noisy. Our algorithms learn to score and rankquantity intervals directly, combining snippet quantity and snippet text information. We report on experiments using hundreds of QCQs with ground truth taken from TREC QA, Wikipedia Infoboxes, and other sources, leading to tens of thousands of candidate snippets and quantities. Our algorithms yield about 20% better MAP and NDCG compared to the best-known collective rankers, and are 35% better than scoring snippets independent of each other.

REFERENCES

• 1: Eugene Agichtein , Luis Gravano, Snowball: extracting relations from large plain-text collections, Proceedings of the fifth ACM conference on Digital libraries, p.85-94, June 02-07, 2000, San Antonio, Texas, United States [doi>10.1145/336597.336644]
• 2: Krisztian Balog , Leif Azzopardi , Maarten de Rijke, A language modeling framework for expert finding, Information Processing and Management: an International Journal, v.45 n.1, p.1-19, January, 2009 [doi>10.1016/j.ipm.2008.06.003]
• 3: Razvan C. Bunescu , Raymond J. Mooney, A shortest path dependency kernel for relation extraction, Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing, p.724-731, October 06-08, 2005, Vancouver, British Columbia, Canada [doi>10.3115/1220575.1220666]
• 4: M. J. Cafarella, C. Re, D. Suciu, O. Etzioni, and M. Banko. Structured querying of web text: A technical challenge. In CIDR, pages 225--234, 2007.
• 5: O. Chapelle, Q. Le, and A. Smola. Large margin optimization of ranking measures. In NIPS 2007 Workshop on Machine Learning for Web Search, 2007.
• 6: Tao Cheng , Xifeng Yan , Kevin Chen-Chuan Chang, EntityRank: searching entities directly and holistically, Proceedings of the 33rd international conference on Very large data bases, September 23-27, 2007, Vienna, Austria
• 7: Charles L. A. Clarke , Gordon V. Cormack , Thomas R. Lynam, Exploiting redundancy in question answering, Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval, p.358-365, September 2001, New Orleans, Louisiana, United States [doi>10.1145/383952.384024]
• 8: Owen de Kretser , Alistair Moffat, Effective document presentation with a locality-based similarity heuristic, Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval, p.113-120, August 15-19, 1999, Berkeley, California, United States [doi>10.1145/312624.312664]
• 9: H. Fang and C. Zhai. Probabilistic models for expert finding. In ECIR, pages 418--430, 2007.
• 10: Thorsten Joachims, Optimizing search engines using clickthrough data, Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, July 23-26, 2002, Edmonton, Alberta, Canada [doi>10.1145/775047.775067]
• 11: T. Joachims, H. Li, T.-Y. Liu, and C. Zhai, editors. Learning to Rank for Information Retrieval, Amsterdam, 2007. SIGIR Workshop.
• 12: Jeongwoo Ko , Eric Nyberg , Luo Si, A probabilistic graphical model for joint answer ranking in question answering, Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, July 23-27, 2007, Amsterdam, The Netherlands [doi>10.1145/1277741.1277801]
• 13: T.-Y. Liu. Learning to rank for information retrieval. Tutorial at SIGIR, 2008.
• 14: T.-Y. Liu, T. Qin, J. Xu, W. Xiong, and H. Li. LETOR: Benchmark dataset for research on learning to rank for information retrieval. In LR4IR Workshop, 2007.
• 15: V. Moriceau. Numerical data integration for cooperative question-answering. In EACL Workshop on Knowledge and Reasoning for Language Processing, pages 42--49, 2006.
• 16: Desislava Petkova , W. Bruce Croft, Proximity-based document representation for named entity retrieval, Proceedings of the sixteenth ACM conference on Conference on information and knowledge management, November 06-10, 2007, Lisbon, Portugal [doi>10.1145/1321440.1321542]
• 17: John Prager , Sarah Luger , Jennifer Chu-Carroll, Type nanotheories: a framework for term comparison, Proceedings of the sixteenth ACM conference on Conference on information and knowledge management, November 06-10, 2007, Lisbon, Portugal [doi>10.1145/1321440.1321538]
• 18: Tao Qin , Tie-Yan Liu , Xu-Dong Zhang , De-Sheng Wang , Wen-Ying Xiong , Hang Li, Learning to rank relational objects and its application to web search, Proceeding of the 17th international conference on World Wide Web, April 21-25, 2008, Beijing, China [doi>10.1145/1367497.1367553]
• 19: Stephen Robertson, A new interpretation of average precision, Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval, July 20-24, 2008, Singapore, Singapore [doi>10.1145/1390334.1390453]
• 20: Fei Wu , Daniel S. Weld, Autonomously semantifying wikipedia, Proceedings of the sixteenth ACM conference on Conference on information and knowledge management, November 06-10, 2007, Lisbon, Portugal [doi>10.1145/1321440.1321449]
• 21: M. Wu and A. Marian. Corroborating answers from multiple web sources. In WebDB: Tenth International Workshop on the Web and Databases, 2007.
• 22: Yisong Yue , Thomas Finley , Filip Radlinski , Thorsten Joachims, A support vector method for optimizing average precision, Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, July 23-27, 2007, Amsterdam, The Netherlands [doi>10.1145/1277741.1277790]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Retrieval models

General Terms:
Algorithms, Experimentation

Keywords:
aggregating evidence from snippets, learning to rank, quantity search