RepWeb: replicated Web with referential integrity

ABSTRACT

Replication of web content, through mirroring of web sites or browsing off-line content, is one of the most used techniques to increase content availability, reduce network bandwidth usage and minimize browsing delays in the world-wide-web.The world-wide-web does not support referential integrity, i.e., broken links do exist. This has been considered, for some years now, one of the most serious problems of the web. This is true in various fields, e.g.: i) if a user pays for some service in the form of web pages, he requires such pages to be reachable all the time, and ii) archived web resources, either scientific, legal or historic, that are still referenced, need to be preserved and remain available.Current approaches to the broken-link problem are not able to preserve referential integrity on the web and, simultaneously, support replication and minimize storage waste due to memory leaks. Some of them also impose specific authoring and management systems. Thus, the limitations of current systems reside in three issues: transparency, completeness and safety.We propose a system, RepWeb, comprised of an application to access and manage replicated web content and an implementation of an acyclic distributed garbage collection algorithm for wide-area replicated memory, that satisfies all these requirements. It supports replication, enforces referential integrity on the web and minimizes storage waste.

REFERENCES

• 1: K. Andrews, F. Kappe, and H. Maurer. The Hyper-G network information systems. J.UCS, 1(4), April 1995.
• 2: George E. Collins, A method for overlapping and erasure of lists, Communications of the ACM, v.3 n.12, p.655-657, Dec. 1960 [doi>10.1145/367487.367501]
• 3: Michael L. Creech, Author-oriented link management, Proceedings of the fifth international World Wide Web conference on Computer networks and ISDN systems, p.1015-1025, May 1996, Paris, France
• 4: Steve Lawrence , David M. Pennock , Gary William Flake , Robert Krovetz , Frans M. Coetzee , Eric Glover , Finn Årup Nielsen , Andries Kruger , C. Lee Giles, Persistence of Web References in Scientific Research, Computer, v.34 n.2, p.26-31, February 2001
• 5: Host Pulse. Broken-link checker, www.hostpulse.com.
• 6: David Ingham , Steve Caughey , Mark Little, Fixing the “broken-link” problem: the W3Objects approach, Computer Networks and ISDN Systems, v.28 n.7-11, p.1255-1268, May 1996
• 7: D. B. Ingham, M. C. Little, S. J. Caughey, and S. K. Shrivastava. W3Objects: Bringing object-oriented technology to the Web. World-Wide Web Journal, 1, 1995.
• 8: F. Kappe. A Scalable Architecture for Maintaining Referential Integrity in Distributed Information Systems. J. UCS, 1(2):84--104, Feb. 1995.
• 9: LinkAlarm. Linkalarm, http://www.linkalarm.com/.
• 10: John McCarthy, Recursive functions of symbolic expressions and their computation by machine, Part I, Communications of the ACM, v.3 n.4, p.184-195, April 1960 [doi>10.1145/367177.367199]
• 11: L. Moreau and N. Gray. A community of agents maintaining link integrity in the world wide Web. In Proc. of the 3rd Int'l Conference on the Practical Applications of Agents and Multi-Agent Systems (PAAM-98), 1998.
• 12: José M. Piquer, Indirect reference counting: a distributed garbage collection algorithm, Proceedings on Parallel architectures and languages Europe : volume I: parallel architectures and algorithms: volume I: parallel architectures and algorithms, p.150-165, June 1991, Eindhoven, The Netherlands
• 13: V. Reich and D. Rosenthal. Lockss: A permanent Web publishing and access system. D-Lib Magazine, 7, 2001.
• 14: Nicolas Richer , Marc Shapiro, The Memory Behavior of the WWW, or The WWW Considered as a Persistent Store, Revised Papers from the 9th International Workshop on Persistent Object Systems, p.161-176, September 06-08, 2000
• 15: D. Rosenthal and V. Reich. Permanent Web publishing. In Freenix Track, Usenix Annual Technical Conference, Usenix, Berkeley, California, June 2000.
• 16: A. Sanchez, L. Veiga, and P. Ferreira. Distributed garbage collection for wide area replicated memory. In Proc. of the 6th USENIX Conference on Object-Oriented Technologies and Systems (COOTS'01), San Antonio (USA), Jan. 2001.
• 17: SyberSystems. Cvsviaftp: Automagic Web site mirroring via ftp. http://www.siber.org/cvs-via-ftp/.
• 18: Xenu's. Linksleuth http://home.snafu.de/tilman/.