Toward harnessing user feedback for machine learning

ABSTRACT

There has been little research into how end users might be able to communicate advice to machine learning systems. If this resource--the users themselves--could somehow work hand-in-hand with machine learning systems, the accuracy of learning systems could be improved and the users' understanding and trust of the system could improve as well. We conducted a think-aloud study to see how willing users were to provide feedback and to understand what kinds of feedback users could give. Users were shown explanations of machine learning predictions and asked to provide feedback to improve the predictions. We found that users had no difficulty providing generous amounts of feedback. The kinds of feedback ranged from suggestions for reweighting of features to proposals for new features, feature combinations, relational features, and wholesale changes to the learning algorithm. The results show that user feedback has the potential to significantly improve machine learning systems, but that learning algorithms need to be extended in several ways to be able to assimilate this feedback.

REFERENCES

• 1: Altendorf, E., Restificar, E., and Dietterich, T. Learning from sparse data by exploiting monotonicity constraints. Conf. Uncertainty in Artificial Intelligence, 2005.
• 2: Daniel Billsus , David M. Hilbert , Dan Maynes-Aminzade, Improving proactive information systems, Proceedings of the 10th international conference on Intelligent user interfaces, January 10-13, 2005, San Diego, California, USA [doi>10.1145/1040830.1040869]
• 3: David M. Blei , Andrew Y. Ng , Michael I. Jordan, Latent dirichlet allocation, The Journal of Machine Learning Research, 3, p.993-1022, 3/1/2003
• 4: Jim Blythe, Task learning by instruction in tailor, Proceedings of the 10th international conference on Intelligent user interfaces, January 10-13, 2005, San Diego, California, USA [doi>10.1145/1040830.1040874]
• 5: Jake D. Brutlag , Christopher Meek, Challenges of the Email Domain for Text Classification, Proceedings of the Seventeenth International Conference on Machine Learning, p.103-110, June 29-July 02, 2000
• 6: Timothy Chklovski , Varun Ratnakar , Yolanda Gil, User interfaces with semi-formal representations: a study of designing argumentation structures, Proceedings of the 10th international conference on Intelligent user interfaces, January 10-13, 2005, San Diego, California, USA [doi>10.1145/1040830.1040864]
• 7: Cohen, W. Learning rules that classify e-mail. AAAI Spring Symp. Information Access, 1996.
• 8: Nilesh Dalvi , Pedro Domingos , Mausam , Sumit Sanghai , Deepak Verma, Adversarial classification, Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, August 22-25, 2004, Seattle, WA, USA [doi>10.1145/1014052.1014066]
• 9: Getoor, L., Friedman, N., Koller, D. and Pfeffer, A. Learning probabilistic relational models. S. Dzeroski and N. Lavrac (eds.) Relational Data Mining. Springer-Verlag, 2001.
• 10: Hart, S. and Staveland, L. Development of a NASA-TLX (Task load index): Results of empirical and theoretical research. P. Hancock and N. Meshkati (eds.), Human Mental Workload, 1988, 139--183.
• 11: Jonathan L. Herlocker , Joseph A. Konstan , John Riedl, Explaining collaborative filtering recommendations, Proceedings of the 2000 ACM conference on Computer supported cooperative work, p.241-250, December 2000, Philadelphia, Pennsylvania, United States [doi>10.1145/358916.358995]
• 12: Cory Kissinger , Margaret Burnett , Simone Stumpf , Neeraja Subrahmaniyan , Laura Beckwith , Sherry Yang , Mary Beth Rosson, Supporting end-user debugging: what do users want to know?, Proceedings of the working conference on Advanced visual interfaces, May 23-26, 2006, Venezia, Italy [doi>10.1145/1133265.1133293]
• 13: Klimt, B. and Yang, Y. The Enron corpus: A new dataset for email classification research. European Conf. Machine Learning 2004, 217--226.
• 14: Henry Lieberman , Ashwani Kumar, Providing Expert Advice by Analogy for On-Line Help, Proceedings of the IEEE/WIC/ACM International Conference on Intelligent Agent Technology, p.26-32, September 19-22, 2005 [doi>10.1109/IAT.2005.120]
• 15: Marx, Z., Rosenstein, M. T., Kaelbling, L. P., Dietterich, T. G. Transfer learning with an ensemble of background tasks. NIPS 2005 Workshop on Transfer Learning.
• 16: Kevin McCarthy , James Reilly , Lorraine McGinty , Barry Smyth, Experiments in dynamic critiquing, Proceedings of the 10th international conference on Intelligent user interfaces, January 10-13, 2005, San Diego, California, USA [doi>10.1145/1040830.1040871]
• 17: George A. Miller, WordNet: a lexical database for English, Communications of the ACM, v.38 n.11, p.39-41, Nov. 1995 [doi>10.1145/219717.219748]
• 18: Brad A. Myers , David A. Weitzman , Andrew J. Ko , Duen H. Chau, Answering why and why not questions in user interfaces, Proceedings of the SIGCHI conference on Human Factors in computing systems, April 22-27, 2006, Montréal, Québec, Canada [doi>10.1145/1124772.1124832]
• 19: Daniel Oblinger , Vittorio Castelli , Lawrence Bergman, Augmentation-based learning: combining observations and user edits for programming-by-demonstration, Proceedings of the 11th international conference on Intelligent user interfaces, January 29-February 01, 2006, Sydney, Australia [doi>10.1145/1111449.1111494]
• 20: Michael J. Pazzani, Representation of electronic mail filtering profiles: a user study, Proceedings of the 5th international conference on Intelligent user interfaces, p.202-206, January 09-12, 2000, New Orleans, Louisiana, United States [doi>10.1145/325737.325843]
• 21: Amit Phalgune , Cory Kissinger , Margaret Burnett , Curtis Cook , Laura Beckwith , Joseph R. Ruthruff, Garbage in, Garbage out? An Empirical Look at Oracle Mistakes by End-User Programmers, Proceedings of the 2005 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC'05), p.45-52, September 20-24, 2005 [doi>10.1109/VLHCC.2005.40]
• 22: Porter, M. An algorithm for suffix stripping. Program, 14(3), 1980, 130--137.
• 23: Pearl Pu , Li Chen, Trust building with explanation interfaces, Proceedings of the 11th international conference on Intelligent user interfaces, January 29-February 01, 2006, Sydney, Australia [doi>10.1145/1111449.1111475]
• 24: Marc Rettig, Prototyping for tiny fingers, Communications of the ACM, v.37 n.4, p.21-27, April 1994 [doi>10.1145/175276.175288]
• 25: Jianqiang Shen , Lida Li , Thomas G. Dietterich , Jonathan L. Herlocker, A hybrid learning system for recognizing user tasks from desktop activities and email messages, Proceedings of the 11th international conference on Intelligent user interfaces, January 29-February 01, 2006, Sydney, Australia [doi>10.1145/1111449.1111473]
• 26: Ian H. Witten , Eibe Frank, Data Mining: Practical Machine Learning Tools and Techniques, Second Edition (Morgan Kaufmann Series in Data Management Systems), Morgan Kaufmann Publishers Inc., San Francisco, CA, 2005
• 27: GuoDong Zhou , Jian Su, Named entity recognition using an HMM-based chunk tagger, Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, July 07-12, 2002, Philadelphia, Pennsylvania [doi>10.3115/1073083.1073163]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Theory and methods

Additional Classification:
  H. Information Systems
  H.1 MODELS AND PRINCIPLES
      H.1.2 User/Machine Systems
          Subjects: Human factors; Human information processing
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Evaluation/methodology

General Terms:
Algorithms, Design, Human Factors

Keywords:
explanations, machine learning, user feedback for learning