Make it fresh, make it quick: searching a network of personal webservers

ABSTRACT

Personal webservers have proven to be a popular means of sharing files and peer collaboration. Unfortunately, the transient availability and rapidly evolving content on such hosts render centralized, crawl-based search indices stale and incomplete. To address this problem, we propose YouSearch, a distributed search application for personal webservers operating within a shared context (e.g., a corporate intranet). With YouSearch, search results are always fast, fresh and complete -- properties we show arise from an architecture that exploits both the extensive distributed resources available at the peer webservers in addition to a centralized repository of summarized network state. YouSearch extends the concept of a shared context within web communities by enabling peers to aggregate into groups and users to search over specific groups. In this paper, we describe the challenges, design, implementation and experiences with a successful intranet deployment of YouSearch.

REFERENCES

• 1: Apache's HTTP Server Project. http://httpd.apache.org/.
• 2: BadBlue -- The P2P File Sharing Web Server. http://www.badblue.com/
• 3: The Gnutella Network. http://www.gnutella.com/.
• 4: Google -- The Web Search Engine. http://www.google.com/.
• 5: Groove Networks, Inc. Desktop Collaboration Software. http://www.groove.net/.
• 6: ht://Dig - Internet Search Engine Software. http://htdig.org/.
• 7: The KaZaa Media Network. http://www.kazaa.com/.
• 8: Mac OS X: Personal Web Sharing. http://www.mac3d.com/.
• 9: LAN-Based web caching for accelerated web access. http://www.mangosoft.com/products/cachelink.
• 10: Microsoft's Personal Web Server and Peer Web Services. http://www.microsoft.com/.
• 11: The Napster Company. http://www.napster.com/.
• 12: SWISH-E: Simple Web Indexing System for Humans-Enhanced. http://swish-e.org/.
• 13: The XDegrees Company. http://www.xdegrees.com/.
• 14: Roberto J. Bayardo Jr. , Rakesh Agrawal , Daniel Gruhl , Amit Somani, YouServ: a web-hosting and content sharing tool for the masses, Proceedings of the 11th international conference on World Wide Web, May 07-11, 2002, Honolulu, Hawaii, USA [doi>10.1145/511446.511492]
• 15: Krishna Bharat, SearchPad: explicit capture of search context to support Web search, Proceedings of the 9th international World Wide Web conference on Computer networks : the international journal of computer and telecommunications netowrking, p.493-501, June 2000, Amsterdam, The Netherlands
• 16: Burton H. Bloom, Space/time trade-offs in hash coding with allowable errors, Communications of the ACM, v.13 n.7, p.422-426, July 1970 [doi>10.1145/362686.362692]
• 17: D. Carmel, E. Amitay, M. Herscovici, Y. Maarek, Y. Petruschka, and A. Soffer.Juru at TREC 10 - Experiments with Index Pruning.In Proc. Text REtrieval Conference (TREC), 2001.
• 18: Francisco Matias Cuenca-Acuna , Thu D. Nguyen, Text-Based Content Search and Retrieval in Ad-hoc P2P Communities, Revised Papers from the NETWORKING 2002 Workshops on Web Engineering and Peer-to-Peer Computing, p.220-234, May 19-24, 2002
• 19: Emma Brunskill, Building Peer-to-Peer Systems with Chord, a Distributed Lookup Service, Proceedings of the Eighth Workshop on Hot Topics in Operating Systems, p.81, May 20-22, 2001
• 20: Cynthia Dwork , Ravi Kumar , Moni Naor , D. Sivakumar, Rank aggregation methods for the Web, Proceedings of the 10th international conference on World Wide Web, p.613-622, May 01-05, 2001, Hong Kong, Hong Kong [doi>10.1145/371920.372165]
• 21: Sitaram Iyer , Antony Rowstron , Peter Druschel, Squirrel: a decentralized peer-to-peer web cache, Proceedings of the twenty-first annual symposium on Principles of distributed computing, July 21-24, 2002, Monterey, California [doi>10.1145/571825.571861]
• 22: Michael Mitzenmacher, Compressed bloom filters, Proceedings of the twentieth annual ACM symposium on Principles of distributed computing, p.144-150, August 2001, Newport, Rhode Island, United States [doi>10.1145/383962.384004]
• 23: Venkata N. Padmanabhan , Kunwadee Sripanidkulchai, The Case for Cooperative Networking, Revised Papers from the First International Workshop on Peer-to-Peer Systems, p.178-190, March 07-08, 2002
• 24: Sylvia Ratnasamy , Paul Francis , Mark Handley , Richard Karp , Scott Schenker, A scalable content-addressable network, Proceedings of the 2001 conference on Applications, technologies, architectures, and protocols for computer communications, p.161-172, August 2001, San Diego, California, United States
• 25: R. Rivest. RFC 1321: The MD5 Message-Digest Algorithm. As Technical report, Network Working Group, 1992.
• 26: Antony I. T. Rowstron , Peter Druschel, Pastry: Scalable, Decentralized Object Location, and Routing for Large-Scale Peer-to-Peer Systems, Proceedings of the IFIP/ACM International Conference on Distributed Systems Platforms Heidelberg, p.329-350, November 12-16, 2001
• 27: Tyron Stading , Petros Maniatis , Mary Baker, Peer-to-Peer Caching Schemes to Address Flash Crowds, Revised Papers from the First International Workshop on Peer-to-Peer Systems, p.203-213, March 07-08, 2002
• 28: Angelos Stavrou , Dan Rubenstein , Sambit Sahu, A Lightweight, Robust P2P System to Handle Flash Crowds, Proceedings of the 10th IEEE International Conference on Network Protocols, p.226-235, November 12-15, 2002
• 29: Y. Xie and D. O'Hallaron. Locality in Search Engine Queries and its Implications for Caching. In Proc. IEEE INFOCOM, 2000.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.4 Systems and Software
          Subjects: Distributed systems

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.4 Systems and Software
          Subjects: Performance evaluation (efficiency and effectiveness); Information networks
  H.4 INFORMATION SYSTEMS APPLICATIONS
      H.4.1 Office Automation
          Subjects: Groupware
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.4 Hypertext/Hypermedia
          Subjects: User issues; Architectures

General Terms:
Algorithms, Human Factors, Performance

Keywords:
P2P, decentralized systems, information communities, intranet search, peer-to-peer networks, web search