Metadata aggregation and "automated digital libraries": a retrospective on the NSDL experience

ABSTRACT

Over three years ago, the Core Integration team of the National Science Digital Library (NSDL) implemented a digital library based on metadata aggregation using Dublin Core and OAI-PMH. The initial expectation was that such low-barrier technologies would be relatively easy to automate and administer. While this architectural choice permitted rapid deployment of a production NSDL, our three years of experience have contradicted our original expectations of easy automation and low people cost. We have learned that alleged "low-barrier" standards are often harder to deploy than expected. In this paper we report on this experience and comment on the general cost, the functionality, and the ultimate effectiveness of this architecture.

REFERENCES

• 1: Arms, C., Dushay, N., Foulonneau, M., et al., Best Practices for Shareable Metadata - DRAFT, http://comm.nsdl.org/download.php/653/ShareableMetadataBestPractices.doc
• 2: Arms, W.Y. Automated Digital Libraries: How Effectively Can Computers Be Used for the Skilled Tasks of Professional Librarianship? D-Lib Magazine, 6 (7/9).
• 3: Arms, W.Y., Dushay, N., Fulker, D.W., et al. A Case Study in Metadata Harvesting: the NSDL. Library Hi Tech, 21 (2).
• 4: Arms, W.Y., Hillmann, D., Lagoze, C., et al. A Spectrum of Interoperability: The Site for Science Prototype for the NSDL. D-Lib Magazine, 8 (1).
• 5: Berners-Lee, T., Fielding, R. and Masinter, L. Uniform Resource Identifiers (URI): Generic Syntax. RFC, 2396, http://www.ietf.org/rfc/rfc2396.txt.
• 6: A. Broder, On the Resemblance and Containment of Documents, Proceedings of the Compression and Complexity of Sequences 1997, p.21, June 11-13, 1997
• 7: Bruce, T.R. and Hillmann, D. The Continuum of Metadata Quality: Defining, Expressing, Exploiting. in Hillmann, D. ed. Metadata in Practice, American Library Association, Chicago, 2004.
• 8: George Buchanan , Annika Hinze, A generic alerting service for digital libraries, Proceedings of the 5th ACM/IEEE-CS joint conference on Digital libraries, June 07-11, 2005, Denver, CO, USA [doi>10.1145/1065385.1065414]
• 9: Cleverdon, C.W. and Keen, E.M. Factors Determining the Performance of Indexing Systems, Vol. 1: Design.
• 10: Cleverdon, C.W. and Keen, E.M. Factors Determining the Performance of Indexing Systems, Vol. 2: Test Results.
• 11: Cole, T.W. and Shreeves, S. Lessons Learned from the Illinois OAI Metadata Harvesting Project. in Hillmann, D. ed. Metadata in Practice, American Library Association, Chicago, 2004.
• 12: Dushay, N. and Hillmann, D., Analyzing Metadata for Effective Use and Re-Use. in DCMI Metadata Conference and Workshop, (Seattle, 2003).
• 13: Dushay, N. and Hillmann, D., NSDL Metadata Primer, http://metamanagement.comm.nsdl.org/outline.html
• 14: Fisher, R., Lugg, R. and Boese, K.C. Cataloging: how to take a business approach. The Bottom Line: Managing Library Finances, 17 (2). 50--54.
• 15: Muriel Foulonneau , Timothy W. Cole , Thomas G. Habing , Sarah L. Shreeves, Using collection descriptions to enhance an aggregation of harvested item-level metadata, Proceedings of the 5th ACM/IEEE-CS joint conference on Digital libraries, June 07-11, 2005, Denver, CO, USA [doi>10.1145/1065385.1065393]
• 16: Hagedorn, K. OAIster: a "no dead ends" OAI service provider. Library Hi Tech, 21 (2). 170--181.
• 17: Halbert, M., Kaczmarek, J. and Hagedorn, K., Findings from the Mellon Metadata Harvesting Initiative. in Research and Advanced Technology for Digital Libraries: 7th European Conference, ECDL 2003, (Trondheim, Norway, 2003), Springer-Verlag.
• 18: Harrison, T.L., Elango, A., Bollen, J., et al. Initial Experiences Re-exporting Duplicate and Similarity Computations with an OAI-PMH Aggregator. arXiv Report, cs.DL/0401001,
• 19: Hillmann, D., Dushay, N. and Phipps, J., Improving Metadata Quality: Augmentation and Recombination. in DC-2004, (Shanghai, China, 2004).
• 20: Carl Lagoze , William Arms , Stoney Gan , Diane Hillmann , Christopher Ingram , Dean Krafft , Richard Marisa , Jon Phipps , John Saylor , Carol Terrizzi , Walter Hoehn , David Millman , James Allan , Sergio Guzman-Lara , Tom Kalt, Core services in the architecture of the national science digital library (NSDL), Proceedings of the 2nd ACM/IEEE-CS joint conference on Digital libraries, July 14-18, 2002, Portland, Oregon, USA [doi>10.1145/544220.544264]
• 21: Lagoze, C., Krafft, D., Jesuroga, S., et al. An Information Network Overlay Architecture for the NSDL. arXiv/CoRR Report, cs.DL/0501080, http://arxiv.org/abs/cs.DL/0501080.
• 22: Lagoze, C., Krafft, D.B., Payette, S., et al. What Is a Digital Library Anyway? Beyond Search and Access in the NSDL. D-Lib Magazine, 11 (11).
• 23: Lagoze, C., Van de Sompel, H., Nelson, M., et al., Guidelines for Aggregators, Caches and Proxies, http://www.openarchives.org/OAI/2.0/guidelines-aggregator.htm
• 24: Mason, J. and Sutton, S., Education Working Group: Draft Proposal, http://dublincore.org/documents/2000/10/05/education-namespace/
• 25: Mitchell, S., Mooney, M., Mason, J., et al. iVia Open Source Virtual Library System. D-Lib Magazine, 9 (1).
• 26: Saylor, J., NSDL COLLECTION DEVELOPMENT POLICY DRAFT v.030715, http://content.comm.nsdl.org/doc_tracker/docs_download.php?id=452
• 27: Shreeves, S., Knutson, E.M., Stvilia, B., et al., Is "Quality" Metadata "Shareable" Metadata? The Implications of Local Metadata Practices for Federated Collections. in ACRL Twelfth National Conference, (Minneapolis, 2005), ALA.
• 28: Warner, S. The OAI Data-Provider Registration and Validation Service. arXiv, cs.DL/0506010, http://arxiv.org/abs/cs.DL/0506010.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.7 Digital Libraries
          Subjects: Collection

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.7 Digital Libraries
          Subjects: Dissemination; Standards; Systems issues

General Terms:
Design, Experimentation, Management, Performance, Reliability

Keywords:
NSDL, OAI-PMH, architecture, interoperability, metadata