The LIVE-project: retrieval experiments based on evaluation viewpoints

ABSTRACT

Besides the operators 'and', 'or' and 'not' the GRIPS retrieval language contains thesaurus — operators to extend the query and truncation — and context-operators for freetext and Boolean searching. In a similar way several other viewpoints and evaluation measures were defined and applied in the retrieval experiment. Under the assumption that the evaluation measure is an interval scale averaging is done by calculating the arithmetic mean.As levels of the experimental factor /8/ the following similarity measures were used: inner product measure, cosine measure, overlap measure, coefficient of Jaccard and Euclidean distance. As situative factors /8/ the number of documents retrieved by GRIPS, the number of descriptiors of the queries, generality and topic of documents were used.The retrieval experiment is not yet finished completely but several results have already been obtained. For example in the average (over 81 queries) for the above defined viewpoint the ranking with the inner product measure does not indicate a significant improvement compared with the GRIPS-output. In the case of the Euclidean distance measure it seems that in the average the user has to inspect less nonrelevant documents. This means an improvement compared with the unordered retrieved set from the GRIPS-output.For more details of the so called 'two-level retrieval process' and further experimental results we refer to the long version of this paper.

REFERENCES

• 1: Karen Sparck Jones, Information Retrieval Experiment, Butterworth-Heinemann, Newton, MA, 1981
• 2: Reiner, U.: Experlrnente lm Geblet des Information Retrieval -- Ueberbllck und Stand der Forechung. TU Berlin, Fachberelch Informatik, I n s t i t u t fuer Angewandte Informatik, LIv~-Berlcht Nr. 6/83
• 3: Peter Bollmann, The normalized recall and related measures, Proceedings of the 6th annual international ACM SIGIR conference on Research and development in information retrieval, June 06-08, 1983, Bethesda, Maryland
• 4: Bollrnann, P.; Welssmann, V.: Probleme der Mittelwertblldung. Deutecher Dokumentartag, Goettingen, 1983, Saur-Verlag, Muenchen
• 5: P. Bollmann, Two axioms for evaluation measures in information retrieval, Proc. of the third joint BCS and ACM symposium on Research and development in information retrieval, p.233-245, November 1984, King's College, Cambridge
• 6: c/ockurn, F.; Relner, U.: Probleme be1 der Planung yon Information Retrieval Experflnenten. Deutecher Dokumentartag, Goettlngen, 1983, 8aur-Verlag, Muenchen
• 7: Jockurn, F.; Welssman, V.: Stru.ktur und Elemente des Information Retrieval Experlmente. 8tudlen zur Klaselflkation, Band 14, Indeksverlag, Frankfurt am Main, 1988
• 8: Friedbert Jochum, On retrieval tests with an inhomogeneous query collection, Proceedings of the 8th annual international ACM SIGIR conference on Research and development in information retrieval, p.215-224, June 05-07, 1985, Montreal, Quebec, Canada [doi>10.1145/253495.253528]
• 9: Welssman, V.E.: Zur Rolle der Statlstlk bel experhnentellen Untereuchungen. TU BerLin, Fachberelch Informatlk, I n s t i t u t fuer Angewandte Informatik, LIVE-Berlcht NR. 8/83
• 10: Kollvad, E.; Relner, U: Eine semantische Analyse der Suchkomponente des Information-Retrleval-8ystems GRIPS. in preparation (LIVE-Bericht Nr. 2/88)
• 11: Cooper, W.S.: Expected Search Length: A Slngle Measure of Retrieval Effectiveness Based on the Weak Ordering Action of Retrieval Systems. American Documentation, Vol. 19, pp 30-41, 1988