Modelling information persistence on the web

ABSTRACT

Models of web data persistency are essential tools for the designof efficient information extraction systems that repeatedlycollect and process the data. This study models the persistence ofweb data through the measurement of URL and content persistenceacross several snapshots of a national community web, collectedfor 3 years. We found that the lifetimes of URLs and contents aremodelled by logarithmic functions. We gathered statistics on thestructure of the web, identified reasons for URL death andcharacterized persistent URLs and contents. The lasting contentstend to be referenced by different URLs during their lifetime,while half of the contents referenced by persistent URLs do notchange.

REFERENCES

• 1: L. Bent , M. Rabinovich , G. M. Voelker , Z. Xiao, Characterization of a large web site population with implications for content delivery, Proceedings of the 13th international conference on World Wide Web, May 17-20, 2004, New York, NY, USA [doi>10.1145/988672.988743]
• 2: Brian E. Brewington , George Cybenko, How dynamic is the Web?, Computer Networks: The International Journal of Computer and Telecommunications Networking, v.33 n.1-6, p.257-276, June 2000
• 3: C. Castillo. E ective Web Crawling. PhD thesis, University of Chile, November 2004.
• 4: Junghoo Cho , Hector Garcia-Molina, The Evolution of the Web and Implications for an Incremental Crawler, Proceedings of the 26th International Conference on Very Large Data Bases, p.200-209, September 10-14, 2000
• 5: Junghoo Cho , Hector Garcia-Molina, Estimating frequency of change, ACM Transactions on Internet Technology (TOIT), v.3 n.3, p.256-290, August 2003 [doi>10.1145/857166.857170]
• 6: Junghoo Cho , Hector Garcia-Molina , Lawrence Page, Efficient crawling through URL ordering, Computer Networks and ISDN Systems, v.30 n.1-7, p.161-172, April 1, 1998
• 7: F. Douglis, A. Feldmann, B. Krishnamurthy, and J. C. Mogul. Rate of change and other metrics: a live study of the world wide web. In USENIX Symposium on Internet Technologies and Systems, 1997.
• 8: Dennis Fetterly , Mark Manasse , Marc Najork , Janet Wiener, A large-scale study of the evolution of web pages, Proceedings of the 12th international conference on World Wide Web, May 20-24, 2003, Budapest, Hungary [doi>10.1145/775152.775246]
• 9: T. A. S. Foundation. Apache HTTP Server Version 1.3: Module mod include, November 2004.
• 10: Daniel Gomes , André L. Santos , Mário J. Silva, Managing duplicates in a web archive, Proceedings of the 2006 ACM symposium on Applied computing, April 23-27, 2006, Dijon, France [doi>10.1145/1141277.1141465]
• 11: Daniel Gomes , Mário J. Silva, Characterizing a national community web, ACM Transactions on Internet Technology (TOIT), v.5 n.3, p.508-531, August 2005 [doi>10.1145/1084772.1084775]
• 12: Allan Heydon , Marc Najork, Mercator: A scalable, extensible Web crawler, World Wide Web, v.2 n.4, p.219-229, 1999 [doi>10.1023/A:1019213109274]
• 13: Wallace Koehler, Web page change and persistence---a four-year longitudinal study, Journal of the American Society for Information Science and Technology, v.53 n.2, p.162-171, January 15, 2002 [doi>10.1002/asi.10018]
• 14: Steve Lawrence , Frans Coetzee , Eric Glover , Gary Flake , David Pennock , Bob Krovetz , Finn Nielsen , Andries Kruger , Lee Giles, Persistence of information on the web: analyzing citations contained in research articles, Proceedings of the ninth international conference on Information and knowledge management, p.235-242, November 06-11, 2000, McLean, Virginia, United States [doi>10.1145/354756.354824]
• 15: J. Markwell and D. W. Brooks. 'link rot' limits the usefulness of web-based educational materials in biochemistry and molecular biology. Biochemistry and Molecular Biology Education, 31(1):69--72, 2003.
• 16: Alexandros Ntoulas , Junghoo Cho , Christopher Olston, What's new on the web?: the evolution of the web from a search engine perspective, Proceedings of the 13th international conference on World Wide Web, May 17-20, 2004, New York, NY, USA [doi>10.1145/988672.988674]
• 17: Diomidis Spinellis, The decay and failures of web references, Communications of the ACM, v.46 n.1, p.71-77, January 2003 [doi>10.1145/602421.602422]

INDEX TERMS

Primary Classification:
  C. Computer Systems Organization
  C.2 COMPUTER-COMMUNICATION NETWORKS
      C.2.5 Local and Wide-Area Networks
          Subjects: Internet (e.g., TCP/IP)

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.7 Digital Libraries
          Subjects: Collection

General Terms:
Experimentation, Management, Measurement

Keywords:
content persistence, tomba, url persistence