Urban web crawling

ABSTRACT

Local search is increasingly becoming a major focus point of research interest. It is a widely-recognized speciality search with a large application area. Its data is usually aggregated from a variety of sources. One as yet largely untapped source of location data is the WWW. Today, the Web does not explicitly reveal its location-relation; rather this information is hidden somewhere within pages' contents. To exploit such location information, we need to find, extract and geo-spatially index relevant Web pages. For an effective retrieval of such content, this paper examines the application of focused Web crawling to the geospatial domain. We describe our approach for a geo-aware focused crawling of urban areas and other regions with a high building density. We present our experimental results that give us insight into spatial Web information such as location density and link distance between topical pages. Our crawls and evaluations back our hypothesis that geospatially focused crawling is suitable for the urban geospatial topic.

REFERENCES

• 1: D. Ahlers and S. Boll. Geospatially Focused Web Crawling. Datenbank-Spektrum, Special Issue Focused Search, 7(23):3--12, 2007.
• 2: D. Ahlers and S. Boll. Location-based Web search. In A. Scharl and K. Tochterman, editors, The Geospatial Web. Springer, London, 2007.
• 3: D. Ahlers and S. Boll. A Web more Geospatial: Insights into the Location Inside. In Web Science Workshop (WSW2008) at WWW2008, Beijing, China, 2008.
• 4: Einat Amitay , Nadav Har'El , Ron Sivan , Aya Soffer, Web-a-where: geotagging web content, Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval, July 25-29, 2004, Sheffield, United Kingdom [doi>10.1145/1008992.1009040]
• 5: Sergey Brin , Lawrence Page, The anatomy of a large-scale hypertextual Web search engine, Computer Networks and ISDN Systems, v.30 n.1-7, p.107-117, April 1, 1998
• 6: Andrei Broder , Ravi Kumar , Farzin Maghoul , Prabhakar Raghavan , Sridhar Rajagopalan , Raymie Stata , Andrew Tomkins , Janet Wiener, Graph structure in the Web, Computer Networks: The International Journal of Computer and Telecommunications Networking, v.33 n.1-6, p.309-320, June 2000
• 7: W. Cai, S. Wang, and Q. Jiang. Address extraction: Extraction of location-based information from the web. In Y. Zhang, K. Tanaka, J. X. Yu, S. Wang, and M. Li, editors, APWeb 2005. Springer, 2005.
• 8: Soumen Chakrabarti , Kunal Punera , Mallela Subramanyam, Accelerated focused crawling through online relevance feedback, Proceedings of the 11th international conference on World Wide Web, May 07-11, 2002, Honolulu, Hawaii, USA [doi>10.1145/511446.511466]
• 9: Soumen Chakrabarti , Martin van den Berg , Byron Dom, Distributed Hypertext Resource Discovery Through Examples, Proceedings of the 25th International Conference on Very Large Data Bases, p.375-386, September 07-10, 1999
• 10: Soumen Chakrabarti , Martin van den Berg , Byron Dom, Focused crawling: a new approach to topic-specific Web resource discovery, Computer Networks: The International Journal of Computer and Telecommunications Networking, v.31 n.11-16, p.1623-1640, May 17, 1999
• 11: Bill Cheswick , Hal Burch , Steve Branigan, Mapping and visualizing the internet, Proceedings of the annual conference on USENIX Annual Technical Conference, p.1-1, June 18-23, 2000, San Diego, California
• 12: Junyan Ding , Luis Gravano , Narayanan Shivakumar, Computing Geographical Scopes of Web Resources, Proceedings of the 26th International Conference on Very Large Data Bases, p.545-556, September 10-14, 2000
• 13: Weizheng Gao , Hyun Chul Lee , Yingbo Miao, Geographically focused collaborative crawling, Proceedings of the 15th international conference on World Wide Web, May 23-26, 2006, Edinburgh, Scotland [doi>10.1145/1135777.1135822]
• 14: Linda L. Hill, Core Elements of Digital Gazetteers: Placenames, Categories, and Footprints, Proceedings of the 4th European Conference on Research and Advanced Technology for Digital Libraries, p.280-290, September 18-20, 2000
• 15: M. Jakob, M. Gromann, D. Nicklas, and B. Mitschang. DCbot: Finding Spatial Information on the Web. In L. Zhou, B. C. Ooi, and X. Meng, editors, DASFAA 2005, pages 779--790. Springer, 2005.
• 16: Maryam Kamvar , Shumeet Baluja, A large scale study of wireless search behavior: Google mobile search, Proceedings of the SIGCHI conference on Human Factors in computing systems, April 22-27, 2006, Montréal, Québec, Canada [doi>10.1145/1124772.1124877]
• 17: Anukool Lakhina , John W. Byers , Mark Crovella , Ibrahim Matta, On the geographic location of internet resources, Proceedings of the 2nd ACM SIGCOMM Workshop on Internet measurment, November 06-08, 2002, Marseille, France [doi>10.1145/637201.637240]
• 18: M. Levene, An Introduction to Search Engines and Web Navigation, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 2005
• 19: A. Markowetz, Y.-Y. Chen, T. Suel, X. Long, and B. Seeger. Design and Implementation of a Geographic Search Engine. In WebDB 2005, pages 19--24, 2005.
• 20: Kevin S. McCurley, Geospatial mapping and navigation of the web, Proceedings of the 10th international conference on World Wide Web, p.221-229, May 01-05, 2001, Hong Kong, Hong Kong [doi>10.1145/371920.372056]
• 21: Yasuhiko Morimoto , Masaki Aono , Michael E. Houle , Kevin S. McCurley, Extracting Spatial Knowledge from the Web, Proceedings of the 2003 Symposium on Applications and the Internet, p.326, January 27-31, 2003
• 22: Marc Najork , Allan Heydon, High-performance web crawling, Handbook of massive data sets, Kluwer Academic Publishers, Norwell, MA, 2002
• 23: Marc Najork , Janet L. Wiener, Breadth-first crawling yields high-quality pages, Proceedings of the 10th international conference on World Wide Web, p.114-118, May 01-05, 2001, Hong Kong, Hong Kong [doi>10.1145/371920.371965]
• 24: M. Sanderson and J. Kohler. Analyzing Geographic Queries. In Proc. of the ACM SIGIR Workshop on Geographic Information Retrieval, Sheffield, UK, 2004.
• 25: Thanh Tin Tang , David Hawking , Nick Craswell , Kathy Griffiths, Focused crawling for both topical relevance and quality of medical information, Proceedings of the 14th ACM international conference on Information and knowledge management, October 31-November 05, 2005, Bremen, Germany [doi>10.1145/1099554.1099583]
• 26: T. T. Tang, D. Hawking, N. Craswell, and R. S. Sankaranarayana. Focused Crawling in Depression Portal Search: A Feasibility Study. In ADCS 2004, pages 2--9, Melbourne, Australia, 2004.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval

General Terms:
Algorithms, Design, Experimentation, Performance

Keywords:
adaptive crawling, focused crawling, geographic web information retrieval, location-based web search, topical search