Searchers' relevance judgments and criteria in evaluating web pages in a learning style perspective

ABSTRACT

The paper presents the results of a case study of searcher's relevance criteria used for assessments of Web pages in a perspective of learning style. 15 test persons participated in the experiments based on two simulated work tasks that provided cover stories to trigger their information needs. Two learning styles were examined: Global and Sequential learners. The study applied eye-tracking for the observation of relevance hot spots on Web pages, learning style index analysis and post-search interviews to gain more in-depth information on relevance behavior.

Findings reveal that with respect to use of graded relevance scores and number of relevance criteria applied per task and test person there are no significant difference between the different styles. Although there differences are detected in the use of relevance criteria between Global and Sequential learners during assessments, they are statistically insignificant. When interviewed in retrospective the resulting profiles tend to become even similar across learning styles but a shift occurs from instant assessments with content features of web pages replacing topicality judgments as predominant relevance criteria.

REFERENCES

• 1: Carol L. Barry, User-defined relevance criteria: an exploratory study, Journal of the American Society for Information Science, v.45 n.3, p.149-159, April 1994 [doi>10.1002/(SICI)1097-4571(199404)45:3<149::AID-ASI5>3.0.CO;2-J]
• 2: Carol L. Barry , Linda Schamber, Users' criteria for relevance evaluation: a cross-situational comparison, Information Processing and Management: an International Journal, v.34 n.2-3, p.219-236, 1998 [doi>10.1016/S0306-4573(97)00078-2]
• 3: Borlund, P. 2000. Experimental components for the evaluation of interactive information retrieval systems. J. of Doc., 56 (1), 71--90
• 4: Pia Borlund, The concept of relevance in IR, Journal of the American Society for Information Science and Technology, v.54 n.10, p.913-925, August 2003 [doi>10.1002/asi.10286]
• 5: Borlund, P. 2003b. The IIR evaluation model: A framework for evaluation of interactive information retrieval systems. Inf. Research, 8(3), paper no. 152. Available: http://informationr.net/ir/8-3/paper152.html.
• 6: Borlund, P. & Ingwersen, P. 1997. The development of a method for the evaluation of interactive information retrieval systems. J. of Doc., (53)3, 225--250.
• 7: Katriina Byström , Kalervo Järvelin, Task complexity affects information seeking and use, Information Processing and Management: an International Journal, v.31 n.2, p.191-213, March/April 1995 [doi>10.1016/0306-4573(94)00041-Z]
• 8: Erica Cosijn , Peter Ingwersen, Dimensions of relevance, Information Processing and Management: an International Journal, v.36 n.4, p.533-550, July 2000 [doi>10.1016/S0306-4573(99)00072-2]
• 9: Ellis, D., Ford, N., & Wood, F. 1992. Hypertext and learning styles. Final Report of a project funded by the Learning technology Unit. Sheffield: Employment Department.
• 10: Ewing, K. 2005. White paper: studying web pages using eye tracking. Tobii Technology, August 2005, p15.
• 11: Felder, R. M. & Spurlin, J. 2005. Applications, reliability and validity of the Index of Learning Styles. Int. J. Eng. Edu., 21(1), 103--112.
• 12: Ford, N., Miller, D. (1996). Gender differences in Internet perception and use. In: Electronic Lib. & Vis. Inf. Res. Papers from the third ELVIRA conference, 30 April 1996, 87--202. London: ASLIB.
• 13: Nigel Ford , David Miller , Nicola Moss, The role of individual differences in Internet searching: an empirical study, Journal of the American Society for Information Science and Technology, v.52 n.12, p.1049-1066, October 2001 [doi>10.1002/asi.1165.abs]
• 14: Enrique Frias-Martinez , Sherry Y. Chen , Xiaohui Liu, Automatic cognitive style identification of digital library users for personalization, Journal of the American Society for Information Science and Technology, v.58 n.2, p.237-251, January 2007 [doi>10.1002/asi.v58:2]
• 15: Ghinea, G. & Chen, Y. 2003. The impact of cognitive styles on perceptual distributed multimedia quality. British J. of Edu. Tech., 34 (4): p. 393--406.
• 16: Peter Ingwersen , Kalervo Järvelin, The Turn: Integration of Information Seeking and Retrieval in Context (The Information Retrieval Series), Springer-Verlag New York, Inc., Secaucus, NJ, 2005
• 17: Thorsten Joachims , Laura Granka , Bing Pan , Helene Hembrooke , Geri Gay, Accurately interpreting clickthrough data as implicit feedback, Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, August 15-19, 2005, Salvador, Brazil [doi>10.1145/1076034.1076063]
• 18: Leader, L. F. & Klein, J. D. 1996. The effects of search tool type and cognitive style on performance during the hypermedia database searches. Edu. Tech. Res. & Dev., 4(1), 24--51.
• 19: Liu, M. & Reed, W. M. 1994. The relationship between the learning strategies and learning styles in hypermedia environment. Comp. in Hum. Behav., 10(4), 419--434.
• 20: Messick, S. 1984. The nature of cognitive styles: Problems and promise in educational practice. Edu. Psych., 19(2), 59--74.
• 21: Ruth A. Palmquist , Kyung-Sun Kim, Cognitive style and on-line database search experience as predictors of web search performance, Journal of the American Society for Information Science, v.51 n.6, p.558-566, April 2000 [doi>10.1002/(SICI)1097-4571(2000)51:6<558::AID-ASI7>3.0.CO;2-9]
• 22: Kai Puolamäki , Jarkko Salojärvi , Eerika Savia , Jaana Simola , Samuel Kaski, Combining eye movements and collaborative filtering for proactive information retrieval, Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, August 15-19, 2005, Salvador, Brazil [doi>10.1145/1076034.1076062]
• 23: Rayner, S. G. 2000. Reconstructing style differences in thinking and learning: profiling learning performance. In: Int. Perspec. on Indiv. Diff. Vol. 1 Cognitive Styles, ed. by R. J. Riding & S. G. Rayner. Stamford, Connecticut: Ablex Publ. Corp., 115--177.
• 24: Riding, R, J., Rayner, S. G. 1998. Cognitive styles and learning strategies. London: David Fulton.
• 25: Linda Schamber , Michael Eisenberg , Michael S. Nilan, A re-examination of relevance: toward a dynamic, situational definition, Information Processing and Management: an International Journal, v.26 n.6, p.755-776, 1990 [doi>10.1016/0306-4573(90)90050-C]
• 26: Salojärvi, J., Kojo, I., Simola J. & Kaski, S. 2003. Can relevance be inferred from eye movements in information retrieval? In: Proc. of WSOM'03, Workshop on Self-Org. Maps: 261--266. Kyushu Institute of Technology, Kitakyushu, Japan.
• 27: Santally, M. I. & Senteni, A. 2005. A learning object approach to personalized web-based instruction. Available: http://www.eurodl.org/materials/contrib/2005/Santally.htm
• 28: Anastasios Tombros , Ian Ruthven , Joemon M. Jose, Searchers' criteria For assessing web pages, Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval, July 28-August 01, 2003, Toronto, Canada [doi>10.1145/860435.860513]
• 29: Pertti Vakkari, Changes in Search Tactics and Relevance Judgements when Preparing a Research Proposal A Summary of the Findings of a Longitudinal Study, Information Retrieval, v.4 n.3-4, p.295-310, September-December 2001
• 30: Valley, K. 1997. Learning styles and courseware design. Ass. Learn. Tech. J., 5(2), 42--51.
• 31: Ellen M. Voorhees, Variations in relevance judgments and the measurement of retrieval effectiveness, Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, p.315-323, August 24-28, 1998, Melbourne, Australia [doi>10.1145/290941.291017]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL

Keywords:
interactive information retrieval, learning styles, relevance assessments, relevance criteria