Geographic web usage estimation by monitoring DNS caches

ABSTRACT

DNS is one of the most actively used distributed databases on earth, accessed by millions of people every day to transparently convert host names into IP addresses and vice versa. In order to improve their performance, DNS servers also keep temporary records of all requested domain names in their cache. While most of the DNS servers are configured to be used by their local users only, there still exist many DNS servers that respond to public queries. Querying these DNS servers reveals the recently visited domains. Exploiting the geographically distributed nature of DNS, one can gather usage statistics ranging from a single DNS server to global scale. In particular, this enables collecting statistics about geographic differences in web browsing behavior between different regions of a country or the world. In this paper, we present methods to identify these public DNS servers, discuss how to effectively crawl them, and describe our algorithm to extract usage estimations from the crawl data. We also evaluate our estimation algorithm using extensive simulations, and finally use our algorithms to crawl 150 U.S. universities for various domains, and explore the effects of location and time on the access rate of these domains.

REFERENCES

• 1: http://www.alexa.com.
• 2: http://www.compete.com.
• 3: http://www.comscore.com.
• 4: http://www.hitwise.com.
• 5: http://www.quantcast.com.
• 6: Junghoo Cho , Hector Garcia-Molina, Estimating frequency of change, ACM Transactions on Internet Technology (TOIT), v.3 n.3, p.256-290, August 2003 [doi>10.1145/857166.857170]
• 7: C. Davis , P. Vixie , T. Goodwin , I. Dickinson, A Means for Expressing Location Information in the Domain Name System, RFC Editor, 1996
• 8: Junyan Ding , Luis Gravano , Narayanan Shivakumar, Computing Geographical Scopes of Web Resources, Proceedings of the 26th International Conference on Very Large Data Bases, p.545-556, September 10-14, 2000
• 9: Edward W. Felten , Michael A. Schneider, Timing attacks on Web privacy, Proceedings of the 7th ACM conference on Computer and communications security, p.25-32, November 01-04, 2000, Athens, Greece [doi>10.1145/352600.352606]
• 10: L. Grangeia. DNS cache snooping: http://www.sysvalue.com/papers/dns-cache-snooping.
• 11: D. Kaminsky. Black Ops Of TCP/IP 2005.5, New Explorations: Large Graphs, Larger Threats. http://www.doxpara.com/Black_Ops_Of_TCPIP_2005.ppt.
• 12: Dan Kaminsky, Explorations in namespace: white-hat hacking across the domain name system, Communications of the ACM, v.49 n.6, June 2006 [doi>10.1145/1132469.1132503]
• 13: P. V. Mockapetris, Domain names: Concepts and facilities, RFC Editor, 1983
• 14: P. V. Mockapetris, Domain names: Implementation specification, RFC Editor, 1983
• 15: P. V. Mockapetris, Domain names - concepts and facilities, RFC Editor, 1987
• 16: P. V. Mockapetris, Domain names - implementation and specification, RFC Editor, 1987
• 17: A. Papoulis and S. U. Pillai. Probability, Random Variables and Stochastic Processes. McGraw-Hill, 4th edition, International edition, 2002.
• 18: Moheeb Abu Rajab , Jay Zarfoss , Fabian Monrose , Andreas Terzis, A multifaceted approach to understanding the botnet phenomenon, Proceedings of the 6th ACM SIGCOMM conference on Internet measurement, October 25-27, 2006, Rio de Janeriro, Brazil [doi>10.1145/1177080.1177086]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.2 DATABASE MANAGEMENT
      H.2.8 Database applications

General Terms:
Measurement

Keywords:
DNS, web access monitoring, web site usage estimation