Data challenges at Yahoo!

ABSTRACT

In this short paper we describe the data that Yahoo! handles, the current trends in Web applications, and the many challenges that this poses for Yahoo! Research. These challenges have led to the development of new data systems and novel data mining techniques.

REFERENCES

• 1: Ricardo Baeza-Yates , Alvaro R. Pereira Jr. , Nivio Ziviani, WIM: An Information Mining Model for the Web, Proceedings of the Third Latin American Web Congress, p.233, October 31-November 02, 2005 [doi>10.1109/LAWEB.2005.50]
• 2: Ricardo Baeza-Yates , Alessandro Tiberi, Extracting semantic relations from query logs, Proceedings of the 13th ACM SIGKDD international conference on Knowledge discovery and data mining, August 12-15, 2007, San Jose, California, USA [doi>10.1145/1281192.1281204]
• 3: Ricardo Baeza-Yates, Peter Mika, and Hugo Zaragoza. Search, Web 2.0, and the Semantic Web. IEEE Intelligent Systems 23, Jan/Feb 2008.
• 4: Community Systems Group, Community systems research at Yahoo!, ACM SIGMOD Record, v.36 n.3, September 2007 [doi>10.1145/1324185.1324195]
• 5: Jeffrey Dean , Sanjay Ghemawat, MapReduce: simplified data processing on large clusters, Proceedings of the 6th conference on Symposium on Opearting Systems Design & Implementation, p.10-10, December 06-08, 2004, San Francisco, CA
• 6: Giuseppe DeCandia , Deniz Hastorun , Madan Jampani , Gunavardhan Kakulapati , Avinash Lakshman , Alex Pilchin , Swaminathan Sivasubramanian , Peter Vosshall , Werner Vogels, Dynamo: amazon's highly available key-value store, Proceedings of twenty-first ACM SIGOPS symposium on Operating systems principles, October 14-17, 2007, Stevenson, Washington, USA
• 7: Sanjay Ghemawat , Howard Gobioff , Shun-Tak Leung, The Google file system, Proceedings of the nineteenth ACM symposium on Operating systems principles, October 19-22, 2003, Bolton Landing, NY, USA
• 8: James Surowiecki, The Wisdom of Crowds, Anchor, 2005
• 9: Usama Fayyad. Interview in SIGKDD by Gregory Piatetsky-Shapiro. http://www.sigkdd.org/explorations/-issues/7-2-2005-12/fayyad.html, 2005
• 10: http://www.vertica.com
• 11: Yahoo! Research Team. Content, Metadata and Behavioral Information: Directions for Yahoo! Research. Bulletin of the IEEE Computer Society Technical Committee on Data Engineering, 2006.
• 12: D. Watts. Six Degrees: The Science of a Connected Age. W. W. Norton & Company, 2003.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.2 DATABASE MANAGEMENT
      H.2.4 Systems

Additional Classification:
  H. Information Systems
  H.2 DATABASE MANAGEMENT
      H.2.8 Database applications
          Subjects: Data mining