A comparative user study on rating vs. personality quiz based preference elicitation methods

ABSTRACT

We conducted a user study evaluating two preference elicitation approaches based on ratings and personality quizzes respectively. Three criteria were used in this comparative study: perceived accuracy, user effort and user loyalty. Results from our study show that the perceived accuracy in two systems is not significantly different. However, users expended significantly less effort, both perceived cognitive effort and actual task time, to complete the preference profile establishing process in the personality quiz-based system than in the rating-based system. Additionally, users expressed stronger intention to reuse the personality quiz-based system and introduce it to their friends. After using these two systems, 53% of users preferred the personality quiz-based system vs. 13% of users preferred the rating-based system, since most users thought the former is easier to use.

REFERENCES

• 1: Gediminas Adomavicius , Alexander Tuzhilin, Toward the Next Generation of Recommender Systems: A Survey of the State-of-the-Art and Possible Extensions, IEEE Transactions on Knowledge and Data Engineering, v.17 n.6, p.734-749, June 2005 [doi>10.1109/TKDE.2005.99]
• 2: Bilgic, M. and Mooney, R.J. Explaining Recommendations: Satisfaction vs. Promotion. Beyond Personalization Workshop, IUI, 2005.
• 3: Li Chen , Pearl Pu, Hybrid critiquing-based recommender systems, Proceedings of the 12th international conference on Intelligent user interfaces, January 28-31, 2007, Honolulu, Hawaii, USA [doi>10.1145/1216295.1216308]
• 4: Ulrike Gretzel , Daniel Fesenmaier, Persuasion in Recommender Systems, International Journal of Electronic Commerce, v.11 n.2, p.81-100, Number 2 / Winter 2006-7 [doi>10.2753/JEC1086-4415110204]
• 5: Gerald Hubl , Valerie Trifts, Consumer Decision Making in Online Shopping Environments: The Effects of Interactive Decision Aids, Marketing Science, v.19 n.1, p.4-21, January 2000 [doi>10.1287/mksc.19.1.4.15178]
• 6: Jone, N. and Pu, P. User Technology Adoption Issues in Recommender Systems. In Proc. of Networking and Electronic Commerce Research Conference (NAEC2007), 2007.
• 7: Kemp, A.E. The Music Temperament: Psychology and Personality of Music1ians. Oxford University Press, New York, 1996.
• 8: Kivetz, R. and Dimonson, I. Earning the right to indulge: Effort as a determination of customer preferences toward frequency program rewards. Journal of Marketing Research, 39(2002), 155--170.
• 9: Kruger J., Wirtz D., Van Boven L. and Altermatt T.W. The effort heuristic. Journal of Experimental Social Psychology, 40, 1(2004), 91--98.
• 10: McNee, S.M., Lam, S.K., Konstan, J.A., Riedal, J. Interfaces for eliciting new user preferences in recommender systems. User Modeling, Johnstown, PA, USA, Springer Verlag(2003) 178--187.
• 11: Kar Yan Tam , Shuk Ying Ho, Web Personalization as a Persuasion Strategy: An Elaboration Likelihood Model Perspective, Information Systems Research, v.16 n.3, p.271-291, September 2005 [doi>10.1287/isre.1050.0058]
• 12: Rentfrow, P.J. and Gosling, S.D. The do re mi's of everyday life: The Structure and Personality Correlates of Music Preferences. Journal of Personality and Social Psychology 84, 6(2003), 1236--1256.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Evaluation/methodology

Additional Classification:
  H. Information Systems
  H.1 MODELS AND PRINCIPLES
      H.1.2 User/Machine Systems
          Subjects: Software psychology
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Interaction styles (e.g., commands, menus, forms, direct manipulation)

General Terms:
Design, Experimentation, Human Factors

Keywords:
personality quiz, preference elicitation, rating-based, recommender systems, user study