On retrieval tests with an inhomogeneous query collection

ABSTRACT

Retrieval tests are often performed with an inhomogeneous query collection, that means certain properties of queries vary within the collection. This paper investigates under which conditions this inhomogeneity may lead to a loss of measurement precision and validity and should be controlled by an experimental design. In particular, interrelationships between the design of a retrieval test, inhomogeneity of the query collection and measurement theoretical properties of the evaluation measure will be discussed. Based on these results, some recommendations for experimental design of retrieval tests are presented.

REFERENCES

• 1: Belkin, N.J., (1981) Ineffable Concepts in Information Retrieval, in:{17}.
• 2: Bollmann, P., (1980) MeBtheoretische Beschreibung von Effektivit~tsmaBen fur IRS mit zwei R~ngen, interner CIS-Report No.13/80, Institut fur Angewandte Informatik, Technische Universit~t Berlin.
• 3: P. Bollmann , V. S. Cherniavsky, Measurement-theoretical investigation of the MZ-metric, Proceedings of the 3rd annual ACM conference on Research and development in information retrieval, p.256-267, June 23-27, 1980, Cambridge, England
• 4: P. Bollmann, Two axioms for evaluation measures in information retrieval, Proc. of the third joint BCS and ACM symposium on Research and development in information retrieval, p.233-245, November 1984, King's College, Cambridge
• 5: Bredenkamp, J., (1980) Theorie und Planun~ psycholo@ischer Experimente, Darmstadt.
• 6: Campbell, D.T. & Stanley, J.C., (1963) Experimental and Quasi- Experimental Designs for Research, Chicago.
• 7: Haas, O.F. & Kraft, D.H., (1984) Experimental and Quasi-Experimental Designs for Research in Information Science, in: Information Processing and Management, 20, No.1-2, 239-241.
• 8: Huck, S.W. & Sutton, C.O., (1975) Some Comments concerning the Use of monotonic Transformations ,to remove the Interaction in Two Factor ANOVA's, in: Educational and Psychological Measurement, 35, 739-791.
• 9: Jochum, F. & Weissmann, V.E., (1984) A Formal Framework for Experimental Evaluation of Information Systems, LiVE-Report No.6/84, Institut fur Angewandte Informati~ Technische Universit~t Berlin.
• 10: John, J.A. & Quenouille, M.H., (1977) Experiments: Desi@n and Analysis, London.
• 11: Keppel, G. & Saufley, W.H. (1980) Introduction to Design and Analysis, San Francisco.
• 12: Lubin, A., (1961) The Interpretation of significant Interactions, in: Educational and Psychological Measurement, 21, 807-817.
• 13: van Rijsbergen, C.J., (1974) Foundations of Evaluation, in: Journal of Documentation, 30, 365-373.
• 14: Roberts, F.S., (1979) Measurement Theory, London.
• 15: Robertson, S.E., (1981) The Methodology of Information Retrieval Experiment, in: {17}.
• 16: Sparck Jones, K., (1981) Retrieval System Tests 1958-1978, in: {17}.
• 17: Karen Sparck Jones, Information Retrieval Experiment, Butterworth-Heinemann, Newton, MA, 1981
• 18: Tague, J.M., (1981) The Pragmatics of Information Retrieval Experimentation, in: {17}.
• 19: Winer, B.J., (1971) Statistical Principles in Experimental Desi@n, Second Edition, Tokyo et al.