Automatically characterizing resource quality for educational digital libraries

ABSTRACT

With the rise of community-generated web content, the need for automatic characterization of resource quality has grown, particularly in the realm of educational digital libraries. We demonstrate how identifying concrete factors of quality for web-based educational resources can make machine learning approaches to automating quality characterization tractable. Using data from several previous studies of quality, we gathered a set of key dimensions and indicators of quality that were commonly identified by educators. We then performed a mixed-method study of digital library curation experts, showing that our characterization of quality captured the subjective processes used by the experts when assessing resource quality for classroom use. Using key indicators of quality selected from a statistical analysis of our expert study data, we developed a set of annotation guidelines and annotated a corpus of 1000 digital resources for the presence or absence of these key quality indicators. Agreement among annotators was high, and initial machine learning models trained from this corpus were able to identify some indicators of quality with as much as an 18% improvement over the baseline.

REFERENCES

• 1: B. Thomas Adler , Luca de Alfaro, A content-driven reputation system for the wikipedia, Proceedings of the 16th international conference on World Wide Web, May 08-12, 2007, Banff, Alberta, Canada [doi>10.1145/1242572.1242608]
• 2: Joshua E. Blumenstock, Size matters: word count as a measure of quality on wikipedia, Proceeding of the 17th international conference on World Wide Web, April 21-25, 2008, Beijing, China [doi>10.1145/1367497.1367673]
• 3: T. Carey and G. L. Hanley. Extending the impact of open educational resources through alignment with pedagogical content knowledge and institutional strategy: Lessons learned from the merlot community experience. In Opening up education: the collective advancement of education through open technology, open content, and open knowledge, chapter 12. MIT Press, 2008.
• 4: CLEANEVAL home page. http://cleaneval.sigwac.org.uk/, Oct. 2008.
• 5: Climate change collection. http://serc.carleton.edu/climatechange/, Oct. 2008.
• 6: M. Custard and T. Sumner. Using machine learning to support quality judgments. D-Lib Magazine, 11(10), Oct. 2005.
• 7: S. de la Chica. Generating Conceptual Knowledge Representations to Support Students Writing Scientific Explanations. PhD thesis, University of Colorado, 2008.
• 8: H. Devaul, A. Diekema, and J. Ostwald. Computer-assisted assignment of educational standards using natural language processing. Unpublished technical report, Digital Learning Sciences, Boulder, CO, 2007.
• 9: Digital library for earth system education. http://www.dlese.org/, Oct. 2008.
• 10: Digital water education library. http://www.csmate.colostate.edu/DWEL/, Jan. 2004.
• 11: DLESE Community Collection (DCC) scope statement. http://www.dlese.org/Metadata/collections/scopes/dcc-scope.php, Oct. 2008.
• 12: Pavel Dmitriev, As we may perceive: finding the boundaries of compound documents on the web, Proceeding of the 17th international conference on World Wide Web, April 21-25, 2008, Beijing, China [doi>10.1145/1367497.1367640]
• 13: D. F. Dufty, D. Mcnamara, M. Louwerse, Z. Cai, and A. C. Graesser. Automatic evaluation of aspects of document quality. In Proceedings of the 22nd annual international conference on Documentation, 2004.
• 14: Nadav Eiron , Kevin S. McCurley, Untangling compound documents on the web, Proceedings of the fourteenth ACM conference on Hypertext and hypermedia, August 26-30, 2003, Nottingham, UK [doi>10.1145/900051.900070]
• 15: K. A. Ericsson and H. A. Simon. Protocol Analysis: Verbal Reports as Data. The MIT Press, revised edition, Apr. 1993.
• 16: B. J. Fogg , Jonathan Marshall , Othman Laraki , Alex Osipovich , Chris Varma , Nicholas Fang , Jyoti Paul , Akshay Rangnekar , John Shon , Preeti Swani , Marissa Treinen, What makes Web sites credible?: a report on a large quantitative study, Proceedings of the SIGCHI conference on Human factors in computing systems, p.61-68, March 2001, Seattle, Washington, United States [doi>10.1145/365024.365037]
• 17: Melody Y. Ivory , Rashmi R. Sinha , Marti A. Hearst, Empirically validated web page design metrics, Proceedings of the SIGCHI conference on Human factors in computing systems, p.53-60, March 2001, Seattle, Washington, United States [doi>10.1145/365024.365035]
• 18: Thorsten Joachims, Making large-scale support vector machine learning practical, Advances in kernel methods: support vector learning, MIT Press, Cambridge, MA, 1999
• 19: P. V. Ogren, P. G. Wetzler, and S. Bethard. ClearTK: A UIMA toolkit for statistical natural language processing. In UIMA for NLP workshop at Language Resources and Evaluation Conference (LREC), 2008.
• 20: René Reitsma , Byron Marshall , Michael Dalton , Martha Cyr, Exploring educational standard alignment: in search of 'relevance', Proceedings of the 8th ACM/IEEE-CS joint conference on Digital libraries, June 16-20, 2008, Pittsburgh PA, PA, USA [doi>10.1145/1378889.1378901]
• 21: Soo Young Rieh, Judgement of information quality and cognitive authority in the Web, Journal of the American Society for Information Science and Technology, v.53 n.2, p.145-161, January 15, 2002 [doi>10.1002/asi.10017.abs]
• 22: Tamara Sumner , Michael Khoo , Mimi Recker , Mary Marlino, Understanding educator perceptions of "quality" in digital libraries, Proceedings of the 3rd ACM/IEEE-CS joint conference on Digital libraries, May 27-31, 2003, Houston, Texas
• 23: Honglei Zeng , Maher A. Alhossaini , Li Ding , Richard Fikes , Deborah L. McGuinness, Computing trust from revision history, Proceedings of the 2006 International Conference on Privacy, Security and Trust: Bridge the Gap Between PST Technologies and Business Services, October 30-November 01, 2006, Markham, Ontario, Canada [doi>10.1145/1501434.1501445]
• 24: Xiaolan Zhu , Susan Gauch, Incorporating quality metrics in centralized/distributed information retrieval on the World Wide Web, Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, p.288-295, July 24-28, 2000, Athens, Greece [doi>10.1145/345508.345602]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.7 Digital Libraries
          Subjects: Standards

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.6 Library Automation
      H.3.7 Digital Libraries
          Subjects: User issues

  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.7 Natural Language Processing
          Subjects: Text analysis
  I.5 PATTERN RECOGNITION
      I.5.4 Applications
          Subjects: Text processing

General Terms:
Algorithms, Human Factors

Keywords:
educational digital library, learning resource, machine learning, quality