Agreeing to disagree: search engines and their public interfaces

ABSTRACT

Google, Yahoo and MSN all provide both web user interfaces (WUIs) and application programming interfaces (APIs) to their collections. Whether building collections of resources or studying the search engines themselves, the search engines request that researchers use their APIs and not "scrape" the WUIs. However, anecdotal evidence suggests the interfaces produce different results. We provide the first in depth quantitative analysis of the results produced by the Google, MSN and Yahoo API and WUI interfaces. We have queried both interfaces for five months and found significant discrepancies between the interfaces in several categories. In general, we found MSN to produce the most consistent results between their two interfaces. Our findings suggest that the API indexes are not older, but they are probably smaller for Google and Yahoo. We also examined how search results decay over time and built predictive models based on the observed decay rates. Based on our findings, it can take over a year for half of the top 10 results to a popular query to be replaced in Google and Yahoo; for MSN it may take only 2-3 months.

REFERENCES

• 1: Ask terms of service, 2006. http://sp.ask.com/en/docs/about/terms_of_service.shtml.
• 2: J. Bar-Ilan. Search engine results over time-A case study on search engine stability. Cybermetrics, 2/3(1), 1998/99.
• 3: Judit Bar-Ilan, Methods for measuring search engine performance over time, Journal of the American Society for Information Science and Technology, v.53 n.4, p.308-319, February 15, 2002 [doi>10.1002/asi.10047]
• 4: Judit Bar-Ilan, Comparing rankings of search results on the web, Information Processing and Management: an International Journal, v.41 n.6, p.1511-1519, December 2005 [doi>10.1016/j.ipm.2005.03.008]
• 5: J. Bar-Ilan. Expectations versus reality-search engine features needed for web research at mid 2005. Cybermetrics, 9(1), 2005.
• 6: J. Bar-Ilan, M. Levene, and M. Mat-Hassan. Dynamics of search engine rankings-A case study. In Proceedings of the 3rd International Workshop on Web Dynamics, May 2004.
• 7: Judit Bar-Ilan , Mazlita Mat-Hassan , Mark Levene, Methods for comparing rankings of search engine results, Computer Networks: The International Journal of Computer and Telecommunications Networking, v.50 n.10, p.1448-1463, 14 July 2006 [doi>10.1016/j.comnet.2005.10.020]
• 8: Ziv Bar-Yossef , Maxim Gurevich, Random sampling from a search engine's index, Proceedings of the 15th international conference on World Wide Web, May 23-26, 2006, Edinburgh, Scotland [doi>10.1145/1135777.1135833]
• 9: Donna Bergmark, Collection synthesis, Proceedings of the 2nd ACM/IEEE-CS joint conference on Digital libraries, July 14-18, 2002, Portland, Oregon, USA [doi>10.1145/544220.544275]
• 10: Donna Bergmark , Carl Lagoze , Alex Sbityakov, Focused Crawls, Tunneling, and Digital Libraries, Proceedings of the 6th European Conference on Research and Advanced Technology for Digital Libraries, p.91-106, September 16-18, 2002
• 11: Krishna Bharat , Andrei Broder, A technique for measuring the relative size and overlap of public Web search engines, Proceedings of the seventh international conference on World Wide Web 7, p.379-388, April 1998, Brisbane, Australia
• 12: Andrei Broder , Marcus Fontura , Vanja Josifovski , Ravi Kumar , Rajeev Motwani , Shubha Nabar , Rina Panigrahy , Andrew Tomkins , Ying Xu, Estimating corpus size via queries, Proceedings of the 15th ACM international conference on Information and knowledge management, November 06-11, 2006, Arlington, Virginia, USA [doi>10.1145/1183614.1183699]
• 13: D. Clinton. Beyond the SOAP search API, Dec. 2006. http://google-code-updates.blogspot.com/2006/12/beyond-soap-search-api.html.
• 14: K. Curran and A. Doherty. Automated broadcast media monitoring using the Google API. In Proceedings of CCNC 2006, volume 2, pages 1098--1102, 2006.
• 15: M. Cutts. GoogleGuy's posts, June 2005. http://www.webmasterworld.com/forum30/29720.htm.
• 16: M. Cutts. Google datacenters. Video, July 31 2006. http://video.google.com/videoplay?docid=8726665066825965913.
• 17: Did-it, Enquiro, and Eyetools uncover search's Golden Triangle, 2005. http://www.enquiro.com/eye-tracking-pr.asp.
• 18: W. Ding and G. Marchionini. A comparative study of web search service performance. In Proceedings of the ASIS Annual Meeting, volume 33, pages 136--142, 1996.
• 19: Ronald Fagin , Ravi Kumar , D. Sivakumar, Comparing Top k Lists, SIAM Journal on Discrete Mathematics, v.17 n.1, p.134-160, 2003 [doi>10.1137/S0895480102412856]
• 20: P. Festa. Google worm targets AOL, Yahoo. Dec. 28 2004. http://news.com.com/Google+worm+targets+AOL%2C+Yahoo/2100-7349_3-5504769.html.
• 21: S. Gauch, G. Wang, and M. Gomez. Profusion: Intelligent fusion from multiple, distributed search engines. Journal of Universal Computer Science, 2(9):637--649, 1996.
• 22: B. Gillette. Google blacklisting researchers? Dec. 14 2004. http://www.emailbattles.com/2005/12/14/virus_aacdehdcic_ei/.
• 23: Google privacy center: Terms of service, 2006. http://www.google.com/terms_of_service.html.
• 24: A. Gulli , A. Signorini, The indexable web is more than 11.5 billion pages, Special interest tracks and posters of the 14th international conference on World Wide Web, May 10-14, 2005, Chiba, Japan [doi>10.1145/1062745.1062789]
• 25: T. G. Habing, T. W. Cole, and W. H. Mischo. Developing a technical registry of OAI data providers. In Proceedings of ECDL '04, pages 400--410, 2004.
• 26: N. Jain, M. Dahlin, and R. Tewari. Using Bloom filters to refine web search results. In Proceedings of the 8th International Workshop on the Web and Databases, 2005.
• 27: Martin Klein , Michael L. Nelson , Juliet Z. Pao, Augmenting OAI-PMH repository holdings using search engine APIs, Proceedings of the 2007 conference on Digital libraries, June 18-23, 2007, Vancouver, BC, Canada [doi>10.1145/1255175.1255285]
• 28: W. Koehler. A longitudinal study of web pages continued: A consideration of document persistence. Information Research, 9(2), 2004.
• 29: M. Koo and H. Skinner. Improving web searches: Case study of quit-smoking web sites for teenagers. Journal of Medical Internet Research, 5(4), Nov. 2003.
• 30: Reiner Kraft , Raymie Stata, Finding Buying Guides with a Web Carnivore, Proceedings of the First Conference on Latin American Web Congress, p.84, November 10-12, 2003
• 31: Steve Lawrence , C. Lee Giles, Accessibility of information on the Web, intelligence, v.11 n.1, p.32-39, Spring 2000 [doi>10.1145/333175.333181]
• 32: The Lycos 50, 2006. http://50.lycos.com/.
• 33: P. Mayr and F. Tosques. Google Web APIs - an instrument for webometric analyses? In Proceedings of the 10th International Conference of the International Society for Scientometrics and Informetrics (ISSI '05), 2005.
• 34: F. McCown. Comparison of search engine interfaces, 2006. http://www.cs.odu.edu/~fmccown/research/se_apis/.
• 35: F. McCown, J. Bollen, and M. L. Nelson. Evaluation of the NSDL and Google for obtaining pedagogical resources. In Proceedings of ECDL '05, pages 344--355, 2005.
• 36: F. McCown, S. Chan, M. L. Nelson, and J. Bollen. The availability and persistence of web references in D-Lib Magazine. In Proceedings of the 5th International Web Archiving Workshop (IWAW '05), Sept. 2005.
• 37: Frank McCown , Xiaoming Liu , Michael L. Nelson , Mohammad Zubair, Search Engine Coverage of the OAI-PMH Corpus, IEEE Internet Computing, v.10 n.2, p.66-73, March 2006 [doi>10.1109/MIC.2006.41]
• 38: M. Moffatt. Yahoo error: Unable to process request at this time - error 999. Feb. 14 2005. http://murraymoffatt.com/software-problem-0011.html.
• 39: MSN terms of service, 2006. http://tou.live.com/en-us/default.aspx.
• 40: MSN Web Search API. http://msdn.microsoft.com/msn/msnsearch/.
• 41: Gautam Pant, Deriving link-context from HTML tag tree, Proceedings of the 8th ACM SIGMOD workshop on Research issues in data mining and knowledge discovery, June 13-13, 2003, San Diego, California [doi>10.1145/882082.882094]
• 42: Gautam Pant , Kostas Tsioutsiouliklis , Judy Johnson , C. Lee Giles, Panorama: extending digital libraries with topical crawlers, Proceedings of the 4th ACM/IEEE-CS joint conference on Digital libraries, June 07-11, 2004, Tuscon, AZ, USA [doi>10.1145/996350.996384]
• 43: Rob Pike , Sean Dorward , Robert Griesemer , Sean Quinlan, Interpreting the data: Parallel analysis with Sawzall, Scientific Programming, v.13 n.4, p.277-298, October 2005
• 44: C. Snelson. Sampling the Web: The development of a custom search tool for research. Library and Information Science Research Electronic Journal, 16(1), Dec. 2005.
• 45: Amanda Spink , Bernard J. Jansen , Chris Blakely , Sherry Koshman, A study of results overlap and uniqueness among major web search engines, Information Processing and Management: an International Journal, v.42 n.5, p.1379-1391, September 2006 [doi>10.1016/j.ipm.2005.11.001]
• 46: K. C. Sua, S. E. Waldren, and T. B. Patrick. Differences inthe effects of filters on health information retrieval from the internet in three languages from three countries: A comparative study. In Proceedings of MEDINFO 2004, 2004.
• 47: M. Thelwall. Can the Web give useful information about commercial uses of scientific research? Online Information Review, 28:120--130, 2004.
• 48: Liwen Vaughan, New measurements for search engine evaluation proposed and tested, Information Processing and Management: an International Journal, v.40 n.4, p.677-691, May 2004 [doi>10.1016/S0306-4573(03)00043-8]
• 49: What's a "supplemental result?" Google Webmaster Help Center, 2006. http://www.google.com/support/webmasters/bin/answer.py?answer=34473.
• 50: Wikipedia: List of basic computer science topics, 2006. http://en.wikipedia.org/wiki/List_of_basic_computer_science_topics.
• 51: Yahoo! Web Search APIs. http://developer.yahoo.net/search/web/.
• 52: Ziming Zhuang , Rohit Wagle , C. Lee Giles, What's there and what's not?: focused crawling for missing documents in digital libraries, Proceedings of the 5th ACM/IEEE-CS joint conference on Digital libraries, June 07-11, 2005, Denver, CO, USA [doi>10.1145/1065385.1065455]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.5 On-line Information Services
          Subjects: Web-based services

General Terms:
Experimentation, Measurement

Keywords:
API, distance measurement, search engine interfaces, search engine results