The importance of manual assessment in link discovery

ABSTRACT

Using a ground truth extracted from the Wikipedia, and a ground truth created through manual assessment, we show that the apparent performance advantage seen in machine learning approaches to link discovery are an artifact of trivial links that are actively rejected by manual assessors.

REFERENCES

• 1: Geva, S., GPX: Ad-Hoc Queries and Automated Link Discovery in the Wikipedia, INEX 2007, pp. 404--416.
• 2: Itakura, K.Y., C.L. Clarke, University of Waterloo at IN-EX2007: Adhoc and Link-the-Wiki Tracks, INEX 2007.
• 3: Jenkinson, D., K.-C. Leung, and A. Trotman, Wikisearching and Wikilinking, preproceedings of INEX 2008.
• 4: Rada Mihalcea , Andras Csomai, Wikify!: linking documents to encyclopedic knowledge, Proceedings of the sixteenth ACM conference on Conference on information and knowledge management, November 06-10, 2007, Lisbon, Portugal [doi>10.1145/1321440.1321475]
• 5: David Milne , Ian H. Witten, Learning to link with wikipedia, Proceeding of the 17th ACM conference on Information and knowledge management, October 26-30, 2008, Napa Valley, California, USA [doi>10.1145/1458082.1458150]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.4 Systems and Software
          Subjects: Information networks

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.4 Systems and Software
          Subjects: Performance evaluation (efficiency and effectiveness)

General Terms:
Documentation, Experimentation, Human Factors, Performance

Keywords:
INEX, assessment, evaluation, link discovery, wikipedia