A few bad votes too many?: towards robust ranking in social media

ABSTRACT

Online social media draws heavily on active reader participation, such as voting or rating of news stories, articles, or responses to a question. This user feedback is invaluable for ranking, filtering, and retrieving high quality content - tasks that are crucial with the explosive amount of social content on the web. Unfortunately, as social media moves into the mainstream and gains in popularity, the quality of the user feedback degrades. Some of this is due to noise, but, increasingly, a small fraction of malicious users are trying to "game the system" by selectively promoting or demoting content for profit, or fun. Hence, an effective ranking of social media content must be robust to noise in the user interactions, and in particular to vote spam. We describe a machine learning based ranking framework for social media that integrates user interactions and content relevance, and demonstrate its effec- tiveness for answer retrieval in a popular community question answering portal. We consider several vote spam attacks, and introduce a method of training our ranker to increase its robustness to some common forms of vote spam attacks. The results of our large-scale experimental evaluation show that our ranker is signifcicantly more robust to vote spam compared to a state-of-the-art baseline as well as the ranker not explicitly trained to handle malicious interactions.

REFERENCES

• 1: Eugene Agichtein , Eric Brill , Susan Dumais, Improving web search ranking by incorporating user behavior information, Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, August 06-11, 2006, Seattle, Washington, USA [doi>10.1145/1148170.1148177]
• 2: Eugene Agichtein , Carlos Castillo , Debora Donato , Aristides Gionis , Gilad Mishne, Finding high-quality content in social media, Proceedings of the international conference on Web search and web data mining, February 11-12, 2008, Palo Alto, California, USA [doi>10.1145/1341531.1341557]
• 3: Jiang Bian , Yandong Liu , Eugene Agichtein , Hongyuan Zha, Finding the right facts in the crowd: factoid question answering over social media, Proceeding of the 17th international conference on World Wide Web, April 21-25, 2008, Beijing, China [doi>10.1145/1367497.1367561]
• 4: Eric Brill , Susan Dumais , Michele Banko, An analysis of the AskMSR question-answering system, Proceedings of the ACL-02 conference on Empirical methods in natural language processing, p.257-264, July 06, 2002 [doi>10.3115/1118693.1118726]
• 5: J. Friedman. Greedy function approximation: a gradient boosting machine. In Ann. Statist., 2001.
• 6: N. Immorlica, K. Jain, M. Mahdian, and K. Talwar. Click fraud resistant methods for learning click-through rates. In Workshop on Internet and Network Economics (WINE), 2005.
• 7: B. J. Jansen. Adversarial information retrieval aspects of sponsored search. In Proc. of the 2nd international workshop on adversarial information retrieval on the web (AIRWeb), 2006.
• 8: Jiwoon Jeon , W. Bruce Croft , Joon Ho Lee, Finding similar questions in large question and answer archives, Proceedings of the 14th ACM international conference on Information and knowledge management, October 31-November 05, 2005, Bremen, Germany [doi>10.1145/1099554.1099572]
• 9: Jiwoon Jeon , W. Bruce Croft , Joon Ho Lee , Soyeon Park, A framework to predict the quality of answers with non-textual features, Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, August 06-11, 2006, Seattle, Washington, USA [doi>10.1145/1148170.1148212]
• 10: Thorsten Joachims, Optimizing search engines using clickthrough data, Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, July 23-26, 2002, Edmonton, Alberta, Canada [doi>10.1145/775047.775067]
• 11: Thorsten Joachims , Laura Granka , Bing Pan , Helene Hembrooke , Geri Gay, Accurately interpreting clickthrough data as implicit feedback, Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, August 15-19, 2005, Salvador, Brazil [doi>10.1145/1076034.1076063]
• 12: Diane Kelly , Jaime Teevan, Implicit feedback for inferring user preference: a bibliography, ACM SIGIR Forum, v.37 n.2, p.18-28, Fall 2003 [doi>10.1145/959258.959260]
• 13: Bhaskar Mehta , Thomas Hofmann , Peter Fankhauser, Lies and propaganda: detecting spam users in collaborative filtering, Proceedings of the 12th international conference on Intelligent user interfaces, January 28-31, 2007, Honolulu, Hawaii, USA [doi>10.1145/1216295.1216307]
• 14: Ahmed Metwally , Divyakant Agrawal , Amr El Abbadi, Detectives: detecting coalition hit inflation attacks in advertising networks streams, Proceedings of the 16th international conference on World Wide Web, May 08-12, 2007, Banff, Alberta, Canada [doi>10.1145/1242572.1242606]
• 15: Paul Heymann , Georgia Koutrika , Hector Garcia-Molina, Fighting Spam on Social Web Sites: A Survey of Approaches and Future Challenges, IEEE Internet Computing, v.11 n.6, p.36-45, November 2007 [doi>10.1109/MIC.2007.125]
• 16: F. Radlinski. Addressing malicious noise in clickthrough data. In Proc. of the 3rd international workshop on adversarial information retrieval on the web (AIRWeb), 2007.
• 17: F. Radlinski and T. Joachims. Minimally invasive randomization for collecting unbiased preferences from clickthrough logs. In Proc. of the National Conference on Artificial Intelligence (AAAI), 2006.
• 18: Qi Su , Dmitry Pavlov , Jyh-Herng Chow , Wendell C. Baker, Internet-scale collection of human-reviewed data, Proceedings of the 16th international conference on World Wide Web, May 08-12, 2007, Banff, Alberta, Canada [doi>10.1145/1242572.1242604]
• 19: E. M. Voorhees. Overview of the TREC 2003 question answering track. In Text REtrieval Conference, 2003.
• 20: Zhaohui Zheng , Keke Chen , Gordon Sun , Hongyuan Zha, A regression framework for learning ranking functions using relative relevance judgments, Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, July 23-27, 2007, Amsterdam, The Netherlands [doi>10.1145/1277741.1277792]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.5 On-line Information Services

General Terms:
Algorithms, Experimentation, Measurement

Keywords:
community question answering, ranking, robustness, social media, vote spam