Harvesting needed to maintain scientific literature online

ABSTRACT

Millions of scientific articles are accessible freely on the web. While some of them are stored in institutional repositories many are made available on personal pages which are exposed to the net's transience. We found that nearly 11% of URLs of PDF documents containing references to life science publications were not accessible within 5 months after being harvested using a search engine's (SE) API. For most of them (8.4%) no SE cache backup could be found. Although we have yet to estimate the exact rate at which the scientific literature disappears and the duration of its disappearance the results so far are a clear indicator that web harvesting is needed to preserve the online scientific literature.

REFERENCES

• 1: Frank McCown , Michael L. Nelson, Search engines and their public interfaces: which apis are the most synchronized?, Proceedings of the 16th international conference on World Wide Web, May 08-12, 2007, Banff, Alberta, Canada [doi>10.1145/1242572.1242763]
• 4: Google blocks academic http://tinyurl.com/2j7alg last retrieved on February 3, 2008

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.7 Digital Libraries

General Terms:
Design, Management, Reliability

Keywords:
availability, preservation, scientific literature