Developing practical automatic metadata assignment and evaluation tools for internet resources

ABSTRACT

This paper describes the development of practical automatic metadata assignment tools to support automatic record creation for virtual libraries, metadata repositories and digital libraries, with particular reference to library-standard metadata. The development process is incremental in nature, and depends upon an automatic metadata evaluation tool to objectively measure its progress. The evaluation tool is based on and informed by the metadata created and maintained by librarian experts at the INFOMINE Project, and uses different metrics to evaluate different metadata fields. In this paper, we describe the form and function of common metadata fields, and identify appropriate performance measures for these fields. The automatic metadata assignment tools in the iVia virtual library software are described, and their performance is measured. Finally, we discuss the limitations of automatic metadata evaluation, and cases where we choose to ignore its evidence in favor of human judgment.

REFERENCES

• 1: Ken Barker , Nadia Cornacchia, Using Noun Phrase Heads to Extract Document Keyphrases, Proceedings of the 13th Biennial Conference of the Canadian Society on Computational Studies of Intelligence: Advances in Artificial Intelligence, p.40-52, May 14-17, 2000
• 2: Cavnar, W. B. and J. M. Trenkle, N-Gram-Based Text Categorization. In Proc. Third Annual Symposium on Document Analysis and Information Retrieval. UNLV Publications/Reprographics, Las Vegas, NV, 1994, 161--175.
• 3: Chakrabarti, S., Roy, S. and Soundalgekar, M. V. Fast and Accurate Text Classification via Multiple Linear Discriminant Projections. In Proc. VLDB. 2002, 658--669
• 4: Chan, L. M. Inter-indexer consistency in subject cataloging. Information Technology and Libraries, 8, 4. 1989, 349--358.
• 5: Ron A. Dolin , Divyakant Agrawal , Amr El Abbadi, Pharos: a scalable distributed architecture for locating heterogeneous information sources, 1998
• 6: Dublin Core Metadata Initiative Dublin Core Metadata Element Set, Version 1.1: Reference Description. 1995-2005. http://dublincore.org/documents/dces/
• 7: Eibe Frank , Gordon W. Paynter , Ian H. Witten , Carl Gutwin , Craig G. Nevill-Manning, Domain-Specific Keyphrase Extraction, Proceedings of the Sixteenth International Joint Conference on Artificial Intelligence, p.668-673, July 31-August 06, 1999
• 8: Frank, E., Hall, M., and Pfahringer B. Locally Weighted Naive Bayes. In Proc. Conf. Uncertainty in Artificial Intelligence (UAI 2003). 2003, 249--256.
• 9: Eibe Frank , Gordon W. Paynter, Predicting library of congress classifications from library of congress subject headings, Journal of the American Society for Information Science and Technology, v.55 n.3, p.214-227, February 2004 [doi>10.1002/asi.10360]
• 10: Godby, C. J. & Stuler, J. The library of congress classification as a knowledge base for automatic subject categorization. In Subject Retrieval in a Network Environment: Papers Presented at an IFLA Satellite Meeting Sponsored by the IFLA Section on Classification and Indexing and IFLA Section of Information Technology, (Dublin, OH.). OCLC, 2001, 14--16.
• 11: Guy, M. Powell, A. and Day, A. Improving the Quality of Metadata in Eprint Archives. Ariadne 38, January 2004. <http://www.ariadne.ac.uk/issue38/guy/>
• 12: Hui Han , C. Lee Giles , Eren Manavoglu , Hongyuan Zha , Zhenyue Zhang , Edward A. Fox, Automatic document metadata extraction using support vector machines, Proceedings of the 3rd ACM/IEEE-CS joint conference on Digital libraries, May 27-31, 2003, Houston, Texas
• 13: Humphreys J. B. K. PhraseRate: An HTML Keyphrase Extractor. Technical report, University of California, Riverside. June 2002. http://infomine.ucr.edu/projects/Keith_Humphrey/PhraseRate/phraserate.pdf <http://i/>
• 14: Internet Assigned Numbers Authority. MIME Media Types. http://www.iana.org/assignments/media-types/
• 15: S. Jones , S. Lundy , G. Paynter, Interactive Document Summarisation Using Automatically Extracted Keyphrases, Proceedings of the 35th Annual Hawaii International Conference on System Sciences (HICSS'02)-Volume 4, p.101.3, January 07-10, 2002
• 16: Steve Jones , Gordon W. Paynter, Human evaluation of Kea, an automatic keyphrasing system, Proceedings of the 1st ACM/IEEE-CS joint conference on Digital libraries, p.148-156, January 2001, Roanoke, Virginia, United States [doi>10.1145/379437.379473]
• 17: Steve Jones , Gordon W. Paynter, Automatic extraction of document keyphrases for use in digital libraries: evaluation and applications, Journal of the American Society for Information Science and Technology, v.53 n.8, p.653-677, August 2002 [doi>10.1002/asi.10068]
• 18: Jones S. and Paynter G. W. An evaluation of document keyphrase sets. Journal of Digital Information, 4, 1. 2003. http://jodi.ecs.soton.ac.uk/Articles/v04/i01/Jones/
• 19: Kedzierski, A. Artur's Auto Annotator. Masters Thesis, Department of Computer Science, University of California, Riverside. 2002.
• 20: Paul Komarek , Andrew Moore, Logistic regression for data mining and high-dimensional classification, 2004
• 21: Larson, R. R. Experiments in automatic library of congress classification. JASIS, 43, 2. 1992, 130--148.
• 22: Library of Congress. SuperLCCS: Library of Congress Classification Schedules combined with additions and changes. Gale Research Inc. 1986-2001.
• 23: Library of Congress Subject Cataloging Division. Library of Congress Subject Headings (24 Ed.). Library of Congress. 2001.
• 24: Chin-Yew Lin , Eduard Hovy, Automatic evaluation of summaries using N-gram co-occurrence statistics, Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology, p.71-78, May 27-June 01, 2003, Edmonton, Canada [doi>10.3115/1073445.1073465]
• 25: Lovins J. B. Development of a Stemming Algorithm. Mechanical Translation and Computational Linguistics, 11. 1968, 22--31.
• 26: Andrew McCallum , Ronald Rosenfeld , Tom M. Mitchell , Andrew Y. Ng, Improving Text Classification by Shrinkage in a Hierarchy of Classes, Proceedings of the Fifteenth International Conference on Machine Learning, p.359-367, July 24-27, 1998
• 27: Mitchell S., Mooney M., Mason J., Paynter G. W., Ruscheinski J., Kedzierski A., Humphreys K. iVia Open Source Virtual Library System. D-Lib Magazine 9, 1. January 2003. http://www.dlib.org/dlib/january03/mitchell/01mitchell.html
• 28: Miguel E. Ruiz , Padmini Srinivasan, Hierarchical neural networks for text categorization (poster abstract), Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval, p.281-282, August 15-19, 1999, Berkeley, California, United States [doi>10.1145/312624.312700]
• 29: Peter D. Turney, Learning Algorithms for Keyphrase Extraction, Information Retrieval, v.2 n.4, p.303-336, May 2000 [doi>10.1023/A:1009976227802]
• 30: Turney, P. D. Coherent Keyphrase Extraction via Web Mining. In Proc. IJCAI. 2003, 434--442.
• 31: Ian H. Witten , David Bainbridge, How to Build a Digital Library, Elsevier Science Inc., New York, NY, 2002
• 32: Witten, I. H. and Frank, E. Data Mining. Morgan Kaufmann, San Francisco, CA. 2000.
• 33: Ozgur Yilmazel , Christina M. Finneran , Elizabeth D. Liddy, Metaextract: an NLP system to automatically assign metadata, Proceedings of the 4th ACM/IEEE-CS joint conference on Digital libraries, June 07-11, 2004, Tuscon, AZ, USA [doi>10.1145/996350.996405]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.6 Library Automation

General Terms:
Algorithms, Measurement, Performance

Keywords:
INFOMINE, automatic metadata assignment, automatic metadata evaluation, iVia, metadata