How evaluator domain expertise affects search result relevance judgments

ABSTRACT

Traditional search evaluation approaches have often relied on domain experts to evaluate results for each query. Unfortunately, the range of topics present in any representative sample of web queries makes it impractical to have expert evaluators for every topic. In this paper, we investigate the effect of using "generalist" evaluators instead of experts in the domain of queries being evaluated. Empirically, we ind that for queries drawn from domains requiring high expertise, (1) generalists tend to give shallow, inaccurate ratings as compared to experts. (2) Further experiments show that generalists disagree on the underlying meaning of these queries significantly more often than experts, and often appear to "give up'' and fall back on surface features such as keyword matching. (3) Finally, by estimating the percentage of "expertise requiring'' queries in a web query sample, we estimate the impact of using generalists, versus the ideal of having domain experts for every "expertise requiring'' query.

REFERENCES

• 1: Anne Aula , Klaus Nordhausen, Modeling successful performance in Web searching, Journal of the American Society for Information Science and Technology, v.57 n.12, p.1678-1693, October 2006 [doi>10.1002/asi.v57:12]
• 2: R. E. Downing, J. L. Moore, and S. W. Brown. The effects and interaction of spatial visualization and domain expertise on information seeking. Computers in Human Behavior, 21(2):195--209, 2004.
• 3: Geoffrey B. Duggan , Stephen J. Payne, Knowledge in the head and on the web: using topic expertise to aid search, Proceeding of the twenty-sixth annual SIGCHI conference on Human factors in computing systems, April 05-10, 2008, Florence, Italy [doi>10.1145/1357054.1357062]
• 4: Donna Harman, Overview of the first TREC conference, Proceedings of the 16th annual international ACM SIGIR conference on Research and development in information retrieval, p.36-47, June 27-July 01, 1993, Pittsburgh, Pennsylvania, United States [doi>10.1145/160688.160692]
• 5: William Hersh , Chris Buckley , T. J. Leone , David Hickam, OHSUMED: an interactive retrieval evaluation and new large test collection for research, Proceedings of the 17th annual international ACM SIGIR conference on Research and development in information retrieval, p.192-201, July 03-06, 1994, Dublin, Ireland
• 6: Christoph Hölscher , Gerhard Strube, Web search behavior of Internet experts and newbies, Computer Networks: The International Journal of Computer and Telecommunications Networking, v.33 n.1-6, p.337-346, June 2000
• 7: Kalervo Järvelin , Jaana Kekäläinen, Cumulated gain-based evaluation of IR techniques, ACM Transactions on Information Systems (TOIS), v.20 n.4, p.422-446, October 2002 [doi>10.1145/582415.582418]
• 8: C. Jenkins, C. L. Corritore, and S. Wiedenbeck. Patterns of information seeking on the web: A qualitative study of domain expertise and web expertise. IT and Society, 1(3):64--89, 2003.
• 9: Pertti Vakkari , Mikko Pennanen , Sami Serola, Changes of search terms and tactics while writing a research proposal A longitudinal case study, Information Processing and Management: an International Journal, v.39 n.3, p.445-463, May 2003 [doi>10.1016/S0306-4573(02)00031-6]
• 10: Barbara M. Wildemuth, The effects of domain knowledge on search tactic formulation, Journal of the American Society for Information Science and Technology, v.55 n.3, p.246-258, February 2004 [doi>10.1002/asi.10367]
• 11: X. Zhang, H. G. B. Anghelescu, and X. Yuan. Domain knowledge, search behavior, and search effectiveness of engineering and science students: an exploratory study. Information Research, 10(2), 2005.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Relevance feedback

General Terms:
Experimentation, Human Factors, Measurement, Reliability

Keywords:
domain expertise, information retrieval, search evaluation