Data quality in web archiving

Subscribe (Full Service)


	Search: The ACM Digital Library The Guide

	Feedback

Data quality in web archiving

Full text

Pdf (673 KB)

Source

International World Wide Web Conference archive
Proceedings of the 3rd workshop on Information credibility on the web table of contents

Madrid, Spain

SESSION: Evaluating credibility of digital resources table of contents

Pages 19-26

Year of Publication: 2009

ISBN:978-1-60558-488-1

Authors

Marc Spaniol	Max-Planck-Institut für Informatik, Saarbrücken, Germany
Dimitar Denev	Max-Planck-Institut für Informatik, Saarbrücken, Germany
Arturas Mazeika	Max-Planck-Institut für Informatik, Saarbrücken, Germany
Gerhard Weikum	Max-Planck-Institut für Informatik, Saarbrücken, Germany
Pierre Senellart	TELECOM Paristech, Paris, France

Sponsor

ACM: Association for Computing Machinery

Publisher

ACM New York, NY, USA

Bibliometrics

Downloads (6 Weeks): 16, Downloads (12 Months): 69, Citation Count: 0

Additional Information:

abstract references index terms collaborative colleagues

Tools and Actions:	Request Permissions Review this Article Save this Article to a Binder Display Formats: BibTeX EndNote ACM Ref

DOI Bookmark:	Use this link to bookmark this Article: http://doi.acm.org/10.1145/1526993.1526999 What is a DOI?

ABSTRACT

Web archives preserve the history of Web sites and have high long-term value for media and business analysts. Such archives are maintained by periodically re-crawling entire Web sites of interest. From an archivist's point of view, the ideal case to ensure highest possible data quality of the archive would be to "freeze" the complete contents of an entire Web site during the time span of crawling and capturing the site. Of course, this is practically infeasible. To comply with the politeness specification of a Web site, the crawler needs to pause between subsequent http requests in order to avoid unduly high load on the site's http server. As a consequence, capturing a large Web site may span hours or even days, which increases the risk that contents collected so far are incoherent with the parts that are still to be crawled. This paper introduces a model for identifying coherent sections of an archive and, thus, measuring the data quality in Web archiving. Additionally, we present a crawling strategy that aims to ensure archive coherence by minimizing the diffusion of Web site captures. Preliminary experiments demonstrate the usefulness of the model and the effectiveness of the strategy.

REFERENCES

Note: OCR errors may be found in this Reference List extracted from the full text article. ACM has opted to expose the complete List rather than only correct and linked references.

	1	A.-L. Barabási. The Physics of the Web. Physics World, 14:33--38, 7 2001.
	2	Ulrik Brandes , Thomas Erlebach, Network Analysis: Methodological Foundations (Lecture Notes in Computer Science), Springer-Verlag New York, Inc., Secaucus, NJ, 2005
	3	Brian E. Brewington , George Cybenko, Keeping Up with the Changing Web, Computer, v.33 n.5, p.52-58, May 2000 [doi>10.1109/2.841784]
	4	Soumen Chakrabarti, Mining the Web: Discovering Knowledge from HyperText Data, Science & Technology Books, 2002
	5	Junghoo Cho , Hector Garcia-Molina, The Evolution of the Web and Implications for an Incremental Crawler, Proceedings of the 26th International Conference on Very Large Data Bases, p.200-209, September 10-14, 2000
	6	Junghoo Cho , Hector Garcia-Molina, Effective page refresh policies for Web crawlers, ACM Transactions on Database Systems (TODS), v.28 n.4, p.390-426, December 2003 [doi>10.1145/958942.958945]
	7	Junghoo Cho , Hector Garcia-Molina, Estimating frequency of change, ACM Transactions on Internet Technology (TOIT), v.3 n.3, p.256-290, August 2003 [doi>10.1145/857166.857170]
	8	Junghoo Cho , Hector Garcia-Molina , Lawrence Page, Efficient crawling through URL ordering, Proceedings of the seventh international conference on World Wide Web 7, p.161-172, April 1998, Brisbane, Australia
	9	James Hendler , Nigel Shadbolt , Wendy Hall , Tim Berners-Lee , Daniel Weitzner, Web science: an interdisciplinary approach to understanding the web, Communications of the ACM, v.51 n.7, July 2008 [doi>10.1145/1364782.1364798]
	10	L. Jäger. Transkriptivität -- zur medialen Logik der kulturellen Semantik (in German). In L. Jäger and G. Stanitzek (eds), Transkribieren -- Medien/Lektüre, pages 19--41. Fink, München, 2002.
	11	R. Klamma and C. Haasler. Wikis as Social Networks: Evolution and Dynamics. In 2nd SNA-KDD Workshop Social Network Mining and Analysis., 2008.
	12	B. Latour. On recalling ANT. In J. Law and J. Hassard (eds), Actor--Network Theory and After, pages 15--25. Oxford, 1999.
	13	Hsin-Tsang Lee , Derek Leonard , Xiaoming Wang , Dmitri Loguinov, IRLbot: scaling to 6 billion pages and beyond, Proceeding of the 17th international conference on World Wide Web, April 21-25, 2008, Beijing, China [doi>10.1145/1367497.1367556]
	14	Mark Levene , Alexandra Poulovassilis, Web Dynamics, SpringerVerlag, 2004
	15	Julien Masanès, Web Archiving, Springer-Verlag New York, Inc., Secaucus, NJ, 2006
	16	G. Mohr, M. Kimpton, M. Stack, and I. Ranitovic. Introduction to Heritrix, an Archival Quality Web Crawler. In 4th intl. Web Archiving Workshop (IWAW'04), 2004.
	17	Marc Najork , Janet L. Wiener, Breadth-first crawling yields high-quality pages, Proceedings of the 10th international conference on World Wide Web, p.114-118, May 01-05, 2001, Hong Kong, Hong Kong [doi>10.1145/371920.371965]
	18	Christopher Olston , Sandeep Pandey, Recrawl scheduling based on information longevity, Proceeding of the 17th international conference on World Wide Web, April 21-25, 2008, Beijing, China [doi>10.1145/1367497.1367557]
	19	G. Pant, P. Srinivasan, and F. Menczer. Web Dynamics -- Adapting to Change in Content, Size, Topology and Use, chapter Crawling the Web, pages 153--178. Springer, 2004.