Situational relevance and task outcome

ABSTRACT

It is explored how precision and relative recall and items retrieved and assessed as useful by medical students for writing course work are associated with the degree to which these items were cited in the work and how they were used, as assessed by teachers. The main finding was that precision and relative recall were not associated with most of the indicators measuring the quality of the course work. The implications of this finding are discussed concerning outcome oriented evaluation of IR.

REFERENCES

• 1: Borlund, P.: Evaluation of interactive information retrieval systems. Åbo Akademi University Press, Åbo (2000)
• 2: Erica Cosijn , Peter Ingwersen, Dimensions of relevance, Information Processing and Management: an International Journal, v.36 n.4, p.533-550, July 2000 [doi>10.1016/S0306-4573(99)00072-2]
• 3: de Bliek, R., Friedman, C. P., Wildemuth, B. M. et al.: Information Retrieved from a Database and the Augmentation of Personal Knowledge. J. Am. Med. Inf. Assoc., 1 (1994) 328--338
• 4: William R. Hersh, Information Retrieval: A Health and Biomedical Perspective, Springer-Verlag New York, Inc., Secaucus, NJ, 2003
• 5: William Hersh, Relevance and retrieval evaluation: perspectives from medicine, Journal of the American Society for Information Science, v.45 n.3, p.201-206, April 1994 [doi>10.1002/(SICI)1097-4571(199404)45:3<201::AID-ASI9>3.0.CO;2-W]
• 6: Hersh, W. R., Crabtree, M. K., Hickam, D. H. et al.: Factors Associated with Success in Searching MEDLINE and Applying Evidence to Answer Clinical Questions. J. Am. Med. Inf. Assoc., 9 (2002) 283--293
• 7: Linda Schamber , Michael Eisenberg , Michael S. Nilan, A re-examination of relevance: toward a dynamic, situational definition, Information Processing and Management: an International Journal, v.26 n.6, p.755-776, 1990 [doi>10.1016/0306-4573(90)90050-C]
• 8: Steve Smithson, Information retrieval evaluation in practice: a case study approach, Information Processing and Management: an International Journal, v.30 n.2, p.205-221, March-April 1994 [doi>10.1016/0306-4573(94)90065-5]
• 9: Louise T. Su, The relevance of recall and precision in user evaluation, Journal of the American Society for Information Science, v.45 n.3, p.207-217, April 1994 [doi>10.1002/(SICI)1097-4571(199404)45:3<207::AID-ASI10>3.0.CO;2-1]
• 10: Vakkari, P.: Task-Based Information Searching. Annu. Rev. Inform. Sci. Technol., 37 (2003) 413--464
• 11: Vakkari, P., & Hakala, N.: Changes in Relevance Criteria and Problem Stages in Task Performance. J. Doc., 56 (2000) 540--562
• 12: Peiling Wang , Marilyn Domas White, A cognitive model of document use during a research project. Study II. Decisions at the reading and citing stages, Journal of the American Society for Information Science, v.50 n.2, p.98-114, Feb. 1999 [doi>10.1002/(SICI)1097-4571(1999)50:2<98::AID-ASI2>3.3.CO;2-C]
• 13: Barbara M. Wildemuth , Ruth de Bliek , Charles P. Friedman , Dean D. File, Medical students' personal knowledge, searching proficiency, and database use in problem solving, Journal of the American Society for Information Science, v.46 n.8, p.590-607, Sept. 1995 [doi>10.1002/(SICI)1097-4571(199509)46:8<590::AID-ASI7>3.0.CO;2-#]
• 14: Barbara M. Wildemuth , Charles P. Friedman , John Keyes , Stephen M. Downs, A longitudinal study of database-assisted problem solving, Information Processing and Management: an International Journal, v.36 n.3, p.445-459, May 2000 [doi>10.1016/S0306-4573(99)00065-5]
• 15: Barbara M. Wildemuth , Charles P. Friedman , Stephen M. Downs, Hypertext versus Boolean access to biomedical information: a comparison of effectiveness, efficiency, and user preferences, ACM Transactions on Computer-Human Interaction (TOCHI), v.5 n.2, p.156-183, June 1998 [doi>10.1145/287675.287677]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval

General Terms:
Experimentation, Human Factors, Measurement