Effects of performance feedback on users' evaluations of an interactive IR system

ABSTRACT

In this study, we seek to understand how providing feedback to users about their performances with an interactive information retrieval (IIR) system impacts their evaluations of that system. Sixty subjects completed three recall-based searching tasks with an experimental IIR system and were asked to evaluate the system after each task and after finishing all three tasks. Before completing the final evaluation, three-fourths of the subjects were provided with feedback about their performances. Subjects were assigned randomly to one of four feedback conditions: a baseline condition where no feedback was provided; an actual feedback condition where subjects were provided with their real performances; and two conditions where subjects were deceived and told that they performed very well or very poorly. Results show that the type of feedback provided significantly affected subjects' system evaluations; most importantly there was a significant difference in subjects' satisfaction ratings before and after feedback was provided in the actual feedback condition.

REFERENCES

• 1: David C. Blair , M. E. Maron, An evaluation of retrieval effectiveness for a full-text document-retrieval system, Communications of the ACM, v.28 n.3, p.289-299, March 1985 [doi>10.1145/3166.3197]
• 2: Borlund, P. (2003). The IIR evaluation model: A framework for evaluation of interactive information retrieval systems. Information Research, 8(3), no. 152.
• 3: Borlund, P. & Ingwersen, P. (1997). The development of a method for the evaluation of interactive information retrieval systems. Journal of Documentation, 53(3), 225--250.
• 4: Cyril Cleverdon, The Cranfield tests on index language devices, Readings in information retrieval, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1997
• 5: Dumais, S. T. & Belkin, N. J. (2005). The TREC Interactive Tracks: Putting the user into search. E. M. Voorhees & D. K. Harman (Eds.) TREC: Experiment and Evaluation in Information Retrieval (pp. 123--153), MIT Press.
• 6: Fiske, D. W. (1982). Convergent-discriminant validation in measurements and research strategies. In D. Brinbirg & L. H. Kidder (Eds.), Forms of Validity in Research (pp. 77--92). San Francisco, CA: Jossey-Bass.
• 7: Försterling, F. (2001). Attribution: An Introduction to Theories, Research and Applications. East Sussex, UK: Psychology Press, Ltd.
• 8: Erik Frøkjær , Morten Hertzum , Kasper Hornbæk, Measuring usability: are effectiveness, efficiency, and satisfaction really correlated?, Proceedings of the SIGCHI conference on Human factors in computing systems, p.345-352, April 01-06, 2000, The Hague, The Netherlands [doi>10.1145/332040.332455]
• 9: David J. Harper , Diane Kelly, Contextual relevance feedback, Proceedings of the 1st international conference on Information interaction in context, October 18-20, 2006, Copenhagen, Denmark [doi>10.1145/1164820.1164847]
• 10: Hersh, W., & Over, P. (1999). TREC-8 interactive track report. In D. Harman and E. M. Voorhees (Eds.), The Eighth Text Retrieval Conference (TREC-8), 57--64.
• 11: Kasper Hornbæk , Effie Lai-Chong Law, Meta-analysis of correlations among usability measures, Proceedings of the SIGCHI conference on Human factors in computing systems, April 28-May 03, 2007, San Jose, California, USA [doi>10.1145/1240624.1240722]
• 12: Kalervo Järvelin , Jaana Kekäläinen, Cumulated gain-based evaluation of IR techniques, ACM Transactions on Information Systems (TOIS), v.20 n.4, p.422-446, October 2002 [doi>10.1145/582415.582418]
• 13: Diane Kelly , David J. Harper , Brian Landau, Questionnaire mode effects in interactive information retrieval experiments, Information Processing and Management: an International Journal, v.44 n.1, p.122-141, January, 2008 [doi>10.1016/j.ipm.2007.02.007]
• 14: George V. Kissel, The effect of computer experience on subjective and objective software usability measures, Conference companion on Human factors in computing systems, p.284-285, May 07-11, 1995, Denver, Colorado, United States [doi>10.1145/223355.223676]
• 15: Youngme Moon , Clifford Nass, Are computers scapegoats?: attributions of responsibility in human-computer interaction, International Journal of Human-Computer Studies, v.49 n.1, p.79-94, July 1998 [doi>10.1006/ijhc.1998.0199]
• 16: Jakob Nielsen , Jonathan Levy, Measuring usability: preference vs. performance, Communications of the ACM, v.37 n.4, p.66-75, April 1994 [doi>10.1145/175276.175282]
• 17: Podsakoff, P. M., MacKenzie, S. B., Lee, J,-Y., & Podsakoff, N. P. (2003). Common method biases in behavioral research: A critical review of the literature and recommended remedies. Journal of Applied Psychology, 88(5), 879--903.
• 18: Tefko Saracevic, Evaluation of evaluation in information retrieval, Proceedings of the 18th annual international ACM SIGIR conference on Research and development in information retrieval, p.138-146, July 09-13, 1995, Seattle, Washington, United States [doi>10.1145/215206.215351]
• 19: Paul Thomas , David Hawking, Evaluation by comparing result sets in context, Proceedings of the 15th ACM international conference on Information and knowledge management, November 06-11, 2006, Arlington, Virginia, USA [doi>10.1145/1183614.1183632]
• 20: Elaine G. Toms , Luanne Freund , Cara Li, WiIRE: the web interactive information retrieval experimentation system prototype, Information Processing and Management: an International Journal, v.40 n.4, p.655-675, May 2004 [doi>10.1016/j.ipm.2003.08.006]
• 21: Ellen M. Voorhees , Donna K. Harman, TREC: Experiment and Evaluation in Information Retrieval (Digital Libraries and Electronic Publishing), The MIT Press, 2005

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL