Take a look at the new version of this page: [ beta version ]. Tell us what you think.

ABSTRACT

We demonstrate a system designed to elicit relative relevance judgments from users to rank images with respect to an image query. The system has been deployed and in use publicly for approximately one year. Furthermore, preference data collected from the users has been made available for research purposes. Further details regarding research on this system is available from Bennett et al. [1].

REFERENCES

• 1: Paul N. Bennett , David Maxwell Chickering , Anton Mityagin, Learning consensus opinion: mining data from a labeling game, Proceedings of the 18th international conference on World wide web, April 20-24, 2009, Madrid, Spain [doi>10.1145/1526709.1526727]
• 2: Cyril W. Cleverdon, The significance of the Cranfield tests on index languages, Proceedings of the 14th annual international ACM SIGIR conference on Research and development in information retrieval, p.3-12, October 13-16, 1991, Chicago, Illinois, United States [doi>10.1145/122860.122861]
• 3: F. Radlinski and T. Joachims. Minimally invasive randomization for collecting unbiased preferences from clickthrough logs. In AAAI 2005, 2005.
• 4: Filip Radlinski , Robert Kleinberg , Thorsten Joachims, Learning diverse rankings with multi-armed bandits, Proceedings of the 25th international conference on Machine learning, p.784-791, July 05-09, 2008, Helsinki, Finland [doi>10.1145/1390156.1390255]
• 5: S. E. Robertson, The probability ranking principle in IR, Readings in information retrieval, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1997
• 6: Luis von Ahn , Laura Dabbish, Labeling images with a computer game, Proceedings of the SIGCHI conference on Human factors in computing systems, p.319-326, April 24-29, 2004, Vienna, Austria [doi>10.1145/985692.985733]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Selection process

General Terms:
Algorithms, Performance

Keywords:
learning preferences, preference judgments