Automatically assessing resource quality for educational digital libraries

ABSTRACT

With the rise of community-generated web content, the need for automatic assessment of resource quality has grown. We demonstrate how developing a concrete characterization of quality for web-based resources can make machine learning approaches to automating quality assessment in the realm of educational digital libraries tractable. Using data from several previous studies of quality, we gathered a set of key dimensions and indicators of quality that were commonly identified by educators. We then performed a mixed-method study of digital library quality experts, showing that our characterization of quality captured the subjective processes used by the experts when assessing resource quality. Using key indicators of quality selected from a statistical analysis of our expert study data, we developed a set of annotation guidelines and annotated a corpus of 1000 digital resources for the presence or absence of the key quality indicators. Agreement among annotators was high, and initial machine learning models trained from this corpus were able to identify some indicators of quality with as much as an 18% improvement over the baseline.

REFERENCES

• 1: B. Thomas Adler , Luca de Alfaro, A content-driven reputation system for the wikipedia, Proceedings of the 16th international conference on World Wide Web, May 08-12, 2007, Banff, Alberta, Canada [doi>10.1145/1242572.1242608]
• 2: S. ann Knight and J. Burn. Developing a framework for assessing information quality on the world wide web. Informing Science Journal, 8:159--172, 2005.
• 3: Joshua E. Blumenstock, Size matters: word count as a measure of quality on wikipedia, Proceeding of the 17th international conference on World Wide Web, April 21-25, 2008, Beijing, China [doi>10.1145/1367497.1367673]
• 4: Sergey Brin , Lawrence Page, The anatomy of a large-scale hypertextual Web search engine, Computer Networks and ISDN Systems, v.30 n.1-7, p.107-117, April 1, 1998
• 5: CLEANEVAL home page. http://cleaneval.sigwac.org.uk/, Oct. 2008.
• 6: Climate change collection. http://serc.carleton.edu/climatechange/, Oct. 2008.
• 7: M. Custard and T. Sumner. Using machine learning to support quality judgments. D-Lib Magazine, 11(10), Oct. 2005.
• 8: S. de la Chica. Generating Conceptual Knowledge Representations to Support Students Writing Scientific Explanations. PhD thesis, University of Colorado, 2008.
• 9: Digital library for earth system education. http://www.dlese.org/, Oct. 2008.
• 10: Digital water education library. http://www.csmate.colostate.edu/DWEL/, Jan. 2004.
• 11: DLESE Community Collection (DCC) scope statement. http://www.dlese.org/Metadata/collections/scopes/dcc-scope.php, Oct. 2008.
• 12: Pavel Dmitriev, As we may perceive: finding the boundaries of compound documents on the web, Proceeding of the 17th international conference on World Wide Web, April 21-25, 2008, Beijing, China [doi>10.1145/1367497.1367640]
• 13: D. F. Dufty, D. Mcnamara, M. Louwerse, Z. Cai, and A. C. Graesser. Automatic evaluation of aspects of document quality. In Proceedings of the 22nd annual international conference on Documentation, 2004.
• 14: Nadav Eiron , Kevin S. McCurley, Untangling compound documents on the web, Proceedings of the fourteenth ACM conference on Hypertext and hypermedia, August 26-30, 2003, Nottingham, UK [doi>10.1145/900051.900070]
• 15: K. A. Ericsson and H. A. Simon. Protocol Analysis: Verbal Reports as Data. The MIT Press, revised edition, Apr. 1993.
• 16: B. J. Fogg , Jonathan Marshall , Othman Laraki , Alex Osipovich , Chris Varma , Nicholas Fang , Jyoti Paul , Akshay Rangnekar , John Shon , Preeti Swani , Marissa Treinen, What makes Web sites credible?: a report on a large quantitative study, Proceedings of the SIGCHI conference on Human factors in computing systems, p.61-68, March 2001, Seattle, Washington, United States [doi>10.1145/365024.365037]
• 17: Melody Y. Ivory , Rashmi R. Sinha , Marti A. Hearst, Empirically validated web page design metrics, Proceedings of the SIGCHI conference on Human factors in computing systems, p.53-60, March 2001, Seattle, Washington, United States [doi>10.1145/365024.365035]
• 18: Thorsten Joachims, Making large-scale support vector machine learning practical, Advances in kernel methods: support vector learning, MIT Press, Cambridge, MA, 1999
• 19: M. J. Kargar, A. R. Ramli, H. Ibrahim, F. Azimzadeh, and S. B. B. M. Noor. Assessing quality of information on the web towards a comprehensive framework. Iranian Journal of Engineering Sciences, 1, 2007.
• 20: P. V. Ogren, P. G. Wetzler, and S. Bethard. ClearTK: A UIMA toolkit for statistical natural language processing. In UIMA for NLP workshop at Language Resources and Evaluation Conference (LREC), 2008.
• 21: T. C. Reeves, J. M. Laey, and M. Marlino. Using technology as cognitive tools: Research and praxis. In Australian Society for Computers in Learning and Tertiary Education (ASCILITE), 1997.
• 22: Soo Young Rieh, Judgement of information quality and cognitive authority in the Web, Journal of the American Society for Information Science and Technology, v.53 n.2, p.145-161, January 15, 2002 [doi>10.1002/asi.10017.abs]
• 23: B. Stvilia and M. B. Twidale. Assessing information quality of a community-based encyclopedia. In Proceedings of the International Conference on Information Quality, pages 442--454, 2005.
• 24: Tamara Sumner , Michael Khoo , Mimi Recker , Mary Marlino, Understanding educator perceptions of "quality" in digital libraries, Proceedings of the 3rd ACM/IEEE-CS joint conference on Digital libraries, May 27-31, 2003, Houston, Texas
• 25: Tamara Sumner , Mary Marlino, Digital libraries and educational practice: a case for new models, Proceedings of the 4th ACM/IEEE-CS joint conference on Digital libraries, June 07-11, 2004, Tuscon, AZ, USA [doi>10.1145/996350.996389]
• 26: Wikipedia:featured article criteria. http://en.wikipedia.org/wiki/Wikipedia:Featured_article_criteria, Oct. 2008.
• 27: Honglei Zeng , Maher A. Alhossaini , Li Ding , Richard Fikes , Deborah L. McGuinness, Computing trust from revision history, Proceedings of the 2006 International Conference on Privacy, Security and Trust: Bridge the Gap Between PST Technologies and Business Services, October 30-November 01, 2006, Markham, Ontario, Canada [doi>10.1145/1501434.1501445]
• 28: Xiaolan Zhu , Susan Gauch, Incorporating quality metrics in centralized/distributed information retrieval on the World Wide Web, Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, p.288-295, July 24-28, 2000, Athens, Greece [doi>10.1145/345508.345602]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.6 Library Automation

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.7 Digital Libraries
          Subjects: Standards; User issues

  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.7 Natural Language Processing
          Subjects: Text analysis
  I.5 PATTERN RECOGNITION
      I.5.4 Applications
          Subjects: Text processing

General Terms:
Algorithms, Human Factors

Keywords:
digital library, digital resource, machine learning, natural language processing, quality