Cluster-based retrieval using language models

ABSTRACT

Previous research on cluster-based retrieval has been inconclusive as to whether it does bring improved retrieval effectiveness over document-based retrieval. Recent developments in the language modeling approach to IR have motivated us to re-examine this problem within this new retrieval framework. We propose two new models for cluster-based retrieval and evaluate them on several TREC collections. We show that cluster-based retrieval can perform consistently across collections of realistic size, and significant improvements over document-based retrieval can be obtained in a fully automatic manner and without relevance information provided by human.

REFERENCES

Note: OCR errors may be found in this Reference List extracted from the full text article. ACM has opted to expose the complete List rather than only correct and linked references.

	1	Allan, J., Carbonell, J., Doddington, G., Yamron, J., and Yang, Y. (1998). Topic detection and tracking pilot study: Final report. In Proceedings of the DARPA Broadcast News Transcription and Understanding Workshop, pp. 194--218.
	2	Croft, W. B. (1980). A model of cluster searching based on classification. Information Systems, Vol. 5, pp. 189--195.
	3	W. Bruce Croft , John Lafferty, Language Modeling for Information Retrieval, Kluwer Academic Publishers, Norwell, MA, 2003
	4	P. Willett , A. El-Hamdouchi, Comparison of hierarchic agglomerative clustering methods for document retrieval, The Computer Journal, v.32 n.3, p.220-227, June 1989 [doi>10.1093/comjnl/32.3.220]
	5	Evans, D.A.; Huettner, A.; Tong, X.; Jansen, P.; & Bennett, J. (1999). Effectiveness of clustering in ad-hoc retrieval. In TREC-7 proceedings, pp. 90--95.
	6	Griffiths, A., Luckhurst, H.C., and Willett, P. (1986). Using interdocument similarity information in document retrieval systems. Journal of the American Society for Information Science, 37, pp. 3--11.
	7	Marti A. Hearst , Jan O. Pedersen, Reexamining the cluster hypothesis: scatter/gather on retrieval results, Proceedings of the 19th annual international ACM SIGIR conference on Research and development in information retrieval, p.76-84, August 18-22, 1996, Zurich, Switzerland [doi>10.1145/243199.243216]
	8	Jardine, N. and van Rijsbergen, C.J. (1971). The use of hierarchical clustering in information retrieval. Information Storage and Retrieval, 7:217--240.
	9	Anton Leuski, Evaluating document clustering for interactive information retrieval, Proceedings of the tenth international conference on Information and knowledge management, October 05-10, 2001, Atlanta, Georgia, USA [doi>10.1145/502585.502592]
	10	Victor Lavrenko , W. Bruce Croft, Relevance based language models, Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval, p.120-127, September 2001, New Orleans, Louisiana, United States [doi>10.1145/383952.383972]
	11	Xiaoyong Liu , W. Bruce Croft, Passage retrieval based on language models, Proceedings of the eleventh international conference on Information and knowledge management, November 04-09, 2002, McLean, Virginia, USA [doi>10.1145/584792.584854]
	12	David R. H. Miller , Tim Leek , Richard M. Schwartz, A hidden Markov model information retrieval system, Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval, p.214-221, August 15-19, 1999, Berkeley, California, United States [doi>10.1145/312624.312680]
	13	Ponte, J. (2001). Is information retrieval anything more than smoothing? In Proceedings of the Workshop on Language Modeling and Information Retrieval, Carnegie Mellon University, Pittsburgh.
	14	Jay M. Ponte , W. Bruce Croft, A language modeling approach to information retrieval, Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, p.275-281, August 24-28, 1998, Melbourne, Australia [doi>10.1145/290941.291008]
	15	Rosenfeld, R. (2000). Two decades of statistical language modeling: where do we go from here? In Proceedings of the IEEE, 88(8), 2000.
	16	K. Sparck Jones , S. Walker , S. E. Robertson, A probabilistic model of information retrieval: development and comparative experiments, Information Processing and Management: an International Journal, v.36 n.6, p.779-808, Nov.06.2000 [doi>10.1016/S0306-4573(00)00015-7]
	17	K. Sparck Jones , S. Walker , S. E. Robertson, A probabilistic model of information retrieval: development and comparative experiments Part 2, Information Processing and Management: an International Journal, v.36 n.6, p.809-840, Nov.06.2000 [doi>10.1016/S0306-4573(00)00016-9]
	18	Spitters, M., and Kraaij, W. (2001). TNO at TDT2001: Language model-based topic detection. In Topic Detection and Tracking Workshop Report.
	19	Anastasios Tombros , Robert Villa , C. J. Van Rijsbergen, The effectiveness of query-specific hierarchic clustering in information retrieval, Information Processing and Management: an International Journal, v.38 n.4, p.559-582, July 2002 [doi>10.1016/S0306-4573(01)00048-6]
	20	C. J. Van Rijsbergen, Information Retrieval, Butterworth-Heinemann, Newton, MA, 1979
	21	van Rijsbergen, C.J. & Croft, W. B. (1975). Document clustering: An evaluation of some experiments with the Cranfield 1400 collection. Information Processing & Management, 11, pp. 171--182.
	22	Ellen M. Voorhees, The cluster hypothesis revisited, Proceedings of the 8th annual international ACM SIGIR conference on Research and development in information retrieval, p.188-196, June 05-07, 1985, Montreal, Quebec, Canada [doi>10.1145/253495.253524]
	23	Ellen Marie Voorhees, The effectiveness and efficiency of agglomerative hierarchic clustering in document retrieval, Cornell University, Ithaca, NY, 1986
	24	Peter Willett, Recent trends in hierarchic document clustering: a critical review, Information Processing and Management: an International Journal, v.24 n.5, p.577-597, 1988 [doi>10.1016/0306-4573(88)90027-1]
	25	Willet, P. (1985). Query specific automatic document classification. International Forum on Information and Documentation, 10(2), pp. 28--32.
	26	Jinxi Xu , W. Bruce Croft, Cluster-based language models for distributed retrieval, Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval, p.254-261, August 15-19, 1999, Berkeley, California, United States [doi>10.1145/312624.312687]
	27	Yamron, J.P., Carp, I., Gillick, L., Lowe, S.A., and van Mulbregt, P. (1999). Topic tracking in a news stream. In Proceedings of the DARPA Broadcast News Workshop.
	28	Yi Zhang , Jamie Callan , Thomas Minka, Novelty and redundancy detection in adaptive filtering, Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval, August 11-15, 2002, Tampere, Finland [doi>10.1145/564376.564393]
	29	ChengXiang Zhai , John Lafferty, Two-stage language models for information retrieval, Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval, August 11-15, 2002, Tampere, Finland [doi>10.1145/564376.564387]