ACM Portal
  Subscribe (Full Service)    Register (Limited Service, Free)    Login
  Search:   The ACM Digital Library   The Guide
  
ACM Home Page
Please provide us with feedback. Feedback
Crawler-Friendly Web Servers
Full text PdfPdf (513 KB)
Source ACM SIGMETRICS Performance Evaluation Review archive
Volume 28 ,  Issue 2  (September 2000) table of contents
Pages: 9 - 14  
Year of Publication: 2000
ISSN:0163-5999
Authors
Onn Brandman  Dept. of Computer Science, Stanford, CA
Junghoo Cho  Dept. of Computer Science, Stanford, CA
Hector Garcia-Molina  Dept. of Computer Science, Stanford, CA
Narayanan Shivakumar  Dept. of Computer Science, Stanford, CA
Publisher
ACM  New York, NY, USA
Bibliometrics
Downloads (6 Weeks): 3,   Downloads (12 Months): 42,   Citation Count: 6
Additional Information:

abstract   cited by   index terms   collaborative colleagues  

Tools and Actions: Review this Article  
DOI Bookmark: Use this link to bookmark this Article: http://doi.acm.org/10.1145/362883.362894
What is a DOI?

ABSTRACT

In this paper we study how to make web servers (e.g., Apache) more crawler friendly. Current web servers offer the same interface to crawlers and regular web surfers, even though crawlers and surfers have very different performance requirements. We evaluate simple and easy-to-incorporate modifications to web servers so that there are significant bandwidth savings. Specifically, we propose that web servers export meta-data archives decribing their content.


CITED BY  6
Luciano Barbosa , Ana Carolina Salgado , Francisco de Carvalho , Jacques Robin , Juliana Freire, Looking at both the present and the past to efficiently update replicas of web content, Proceedings of the 7th annual ACM international workshop on Web information and data management, November 04-04, 2005, Bremen, Germany
Junghoo Cho , Hector Garcia-Molina , Taher Haveliwala , Wang Lam , Andreas Paepcke , Sriram Raghavan , Gary Wesley, Stanford WebBase components and applications, ACM Transactions on Internet Technology (TOIT), v.6 n.2, p.153-186, May 2006
Michael L. Nelson , Joan A. Smith , Ignacio Garcia del Campo, Efficient, automatic web resource harvesting, Proceedings of the eighth ACM international workshop on Web information and data management, November 10-10, 2006, Arlington, Virginia, USA
Qingzhao Tan , Prasenjit Mitra , C. Lee Giles, Designing clustering-based web crawling policies for search engine crawlers, Proceedings of the sixteenth ACM conference on Conference on information and knowledge management, November 06-10, 2007, Lisbon, Portugal
Rui Cai , Jiang-Ming Yang , Wei Lai , Yida Wang , Lei Zhang, iRobot: an intelligent crawler for web forums, Proceeding of the 17th international conference on World Wide Web, April 21-25, 2008, Beijing, China
Uri Schonfeld , Narayanan Shivakumar, Sitemaps: above and beyond the crawl of duty, Proceedings of the 18th international conference on World wide web, April 20-24, 2009, Madrid, Spain

INDEX TERMS

Primary Classification:
  C. Computer Systems Organization
  C.2 COMPUTER-COMMUNICATION NETWORKS
      C.2.5 Local and Wide-Area Networks
          Subjects: Internet (e.g., TCP/IP)

Additional Classification:
  C. Computer Systems Organization
  C.5 COMPUTER SYSTEM IMPLEMENTATION


General Terms:
Design, Measurement, Theory

Collaborative Colleagues:
Onn Brandman: colleagues
Junghoo Cho: colleagues
Hector Garcia-Molina: colleagues
Narayanan Shivakumar: colleagues

The ACM Portal is published by the Association for Computing Machinery. Copyright © 2009 ACM, Inc.
Terms of Usage   Privacy Policy   Code of Ethics   Contact Us

Useful downloads: Adobe Acrobat    QuickTime    Windows Media Player    Real Player