Integrating rich user feedback into intelligent user interfaces

ABSTRACT

The potential for machine learning systems to improve via a mutually beneficial exchange of information with users has yet to be explored in much detail. Previously, we found that users were willing to provide a generous amount of rich feedback to machine learning systems, and that the types of some of this rich feedback seem promising for assimilation by machine learning algorithms. Following up on those findings, we ran an experiment to assess the viability of incorporating real-time keyword-based feedback in initial training phases when data is limited. We found that rich feedback improved accuracy but an initial unstable period often caused large fluctuations in classifier behavior. Participants were able to give feedback by relying heavily on system communication in order to respond to changes. The results show that in order to benefit from the user's knowledge, machine learning systems must be able to absorb keyword-based rich feedback in a graceful manner and provide clear explanations of their predictions.

REFERENCES

• 1: Altendorf, E., Restificar, E., Dietterich, T. Learning from sparse data by exploiting monotonicity constraints. Proc. UAI (2005).
• 2: Laura Beckwith , Cory Kissinger , Margaret Burnett , Susan Wiedenbeck , Joseph Lawrance , Alan Blackwell , Curtis Cook, Tinkering and gender in end-user programmers' debugging, Proceedings of the SIGCHI conference on Human Factors in computing systems, April 22-27, 2006, Montréal, Québec, Canada [doi>10.1145/1124772.1124808]
• 3: Daniel Billsus , David M. Hilbert , Dan Maynes-Aminzade, Improving proactive information systems, Proceedings of the 10th international conference on Intelligent user interfaces, January 10-13, 2005, San Diego, California, USA [doi>10.1145/1040830.1040869]
• 4: Jim Blythe, Task learning by instruction in tailor, Proceedings of the 10th international conference on Intelligent user interfaces, January 10-13, 2005, San Diego, California, USA [doi>10.1145/1040830.1040874]
• 5: Avrim Blum , Tom Mitchell, Combining labeled and unlabeled data with co-training, Proceedings of the eleventh annual conference on Computational learning theory, p.92-100, July 24-26, 1998, Madison, Wisconsin, United States [doi>10.1145/279943.279962]
• 6: Jake D. Brutlag , Christopher Meek, Challenges of the Email Domain for Text Classification, Proceedings of the Seventeenth International Conference on Machine Learning, p.103-110, June 29-July 02, 2000
• 7: Byrnes, J. P., Miller, D. C., Schafer W. D. Gender differences in risk taking: A meta-analysis. Psychological Bulletin 125 (1999), 367--383.
• 8: Chapelle, O., Scholkopf, B., Zien, A. Semi-Supervised Learning. MIT Press, Cambridge, MA, 2006.
• 9: Cohen, W. Learning rules that classify e-mail. Proc. AAAI Spring Symp. Information Access (1996).
• 10: Elisabeth Crawford , Judy Kay , Eric McCreath, IEMS - The Intelligent Email Sorter, Proceedings of the Nineteenth International Conference on Machine Learning, p.83-90, July 08-12, 2002
• 11: Aron Culotta , Trausti Kristjansson , Andrew McCallum , Paul Viola, Corrective feedback and persistent learning for information extraction, Artificial Intelligence, v.170 n.14, p.1101-1122, October 2006 [doi>10.1016/j.artint.2006.08.001]
• 12: Allen Cypher , Daniel C. Halbert , David Kurlander , Henry Lieberman , David Maulsby , Brad A. Myers , Alan Turransky, Watch what I do: programming by demonstration, MIT Press, Cambridge, MA, 1993
• 13: Facione, P. A. The California Critical Thinking Skills Test: College Level Technical Report #2. California Academic Press, Millbrae CA. 1990 (ERIC Document Reproduction Service No. ED 327 550).
• 14: Jerry Alan Fails , Dan R. Olsen, Jr., Interactive machine learning, Proceedings of the 8th international conference on Intelligent user interfaces, January 12-15, 2003, Miami, Florida, USA [doi>10.1145/604045.604056]
• 15: Fung, G., Mangasarian, O., Shavlik, J. Knowledge-based support vector machine classifiers. Proc. NIPS (2002).
• 16: Hart, S., Staveland, L. Development of a NASA-TLX (Task load index): Results of empirical and theoretical research, Human Mental Workload (1988), Hancock, P. and Meshkati, N. (eds.), 139--183.
• 17: Jonathan L. Herlocker , Joseph A. Konstan , John Riedl, Explaining collaborative filtering recommendations, Proceedings of the 2000 ACM conference on Computer supported cooperative work, p.241-250, December 2000, Philadelphia, Pennsylvania, United States [doi>10.1145/358916.358995]
• 18: Karen Holtzblatt , Hugh Beyer, Making customer-centered design work for teams, Communications of the ACM, v.36 n.10, p.92-103, Oct. 1993 [doi>10.1145/163430.164050]
• 19: Yifen Huang , Tom M. Mitchell, Text clustering with extended user feedback, Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, August 06-11, 2006, Seattle, Washington, USA [doi>10.1145/1148170.1148242]
• 20: Lieberman, H., (ed.) Your Wish is My Command: Programming By Example. 2001.
• 21: Liu, B. Li, X. Lee, W., Yu, P. Text Classification by Labeling Words. Proc. AAAI (2004).
• 22: Kevin McCarthy , James Reilly , Lorraine McGinty , Barry Smyth, Experiments in dynamic critiquing, Proceedings of the 10th international conference on Intelligent user interfaces, January 10-13, 2005, San Diego, California, USA [doi>10.1145/1040830.1040871]
• 23: Richard G. McDaniel , Brad A. Myers, Getting more out of programming-by-demonstration, Proceedings of the SIGCHI conference on Human factors in computing systems: the CHI is the limit, p.442-449, May 15-20, 1999, Pittsburgh, Pennsylvania, United States [doi>10.1145/302979.303127]
• 24: Daniel Oblinger , Vittorio Castelli , Lawrence Bergman, Augmentation-based learning: combining observations and user edits for programming-by-demonstration, Proceedings of the 11th international conference on Intelligent user interfaces, January 29-February 01, 2006, Sydney, Australia [doi>10.1145/1111449.1111494]
• 25: Amit Phalgune , Cory Kissinger , Margaret Burnett , Curtis Cook , Laura Beckwith , Joseph R. Ruthruff, Garbage in, Garbage out? An Empirical Look at Oracle Mistakes by End-User Programmers, Proceedings of the 2005 IEEE Symposium on Visual Languages and Human-Centric Computing, p.45-52, September 20-24, 2005 [doi>10.1109/VLHCC.2005.40]
• 26: Jianqiang Shen , Lida Li , Thomas G. Dietterich , Jonathan L. Herlocker, A hybrid learning system for recognizing user tasks from desktop activities and email messages, Proceedings of the 11th international conference on Intelligent user interfaces, January 29-February 01, 2006, Sydney, Australia [doi>10.1145/1111449.1111473]
• 27: Simone Stumpf , Vidya Rajaram , Lida Li , Margaret Burnett , Thomas Dietterich , Erin Sullivan , Russell Drummond , Jonathan Herlocker, Toward harnessing user feedback for machine learning, Proceedings of the 12th international conference on Intelligent user interfaces, January 28-31, 2007, Honolulu, Hawaii, USA [doi>10.1145/1216295.1216316]
• 28: Stumpf S, Rajaram V, Li L, Wong W-K, Burnett M, Dietterich T, Sullivan E, Herlocker J. Interacting Meaningfully with Machine Learning Systems: Three Experiments. EECS OSU Technical Report 2007-46, http://eecs.oregonstate.edu/library.
• 29: Malcolm Ware , Eibe Frank , Geoffrey Holmes , Mark Hall , Ian H. Witten, Interactive machine learning: letting users build classifiers, International Journal of Human-Computer Studies, v.56 n.3, p.281-292, September, 2002

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)

Additional Classification:
  H. Information Systems
  H.1 MODELS AND PRINCIPLES
      H.1.2 User/Machine Systems

Keywords:
machine learning, user feedback