Overview based example selection in end user interactive concept learning

ABSTRACT

Interaction with large unstructured datasets is difficult because existing approaches, such as keyword search, are not always suited to describing concepts corresponding to the distinctions people want to make within datasets. One possible solution is to allow end users to train machine learning systems to identify desired concepts, a strategy known as interactive concept learning. A fundamental challenge is to design systems that preserve end user flexibility and control while also guiding them to provide examples that allow the machine learning system to effectively learn the desired concept. This paper presents our design and evaluation of four new overview based approaches to guiding example selection. We situate our explorations within CueFlik, a system examining end user interactive concept learning in Web image search. Our evaluation shows our approaches not only guide end users to select better training examples than the best performing previous design for this application, but also reduce the impact of not knowing when to stop training the system. We discuss challenges for end user interactive concept learning systems and identify opportunities for future research on the effective design of such systems.

REFERENCES

• 1: Baum, E.B. and Lang, K. Query Learning can work Poorly when a Human Oracle is Used. Proceedings of Neural Networks 1992.
• 2: Blum, A., Chalasani, P., Goldman, S.A. and Slonim, D.K. Learning with Unreliable Boundary Queries. Proceedings of COLT 1995, pp. 98--107.
• 3: Bunt, A., Conati, C. and McGrenere, J. Supporting Interface Tailoring Using a Mixed-Initiative Approach. Proceedings of IUI 2007, pp. 92--101.
• 4: Chen, F., Gargi, U., Niles, L., and Schuetze, H. Multi-Modal Browsing of Images in Web Documents. Proceedings of SPIE Document Recognition and Retrieval VI, 3651 (1999), pp. 122--133.
• 5: Cutting, D.R., Karger, D.R., Pedersen, J.O., and Tukey, J.W. Scatter/Gather: A Cluster-Based Approach to Browsing Large Document Collections. Proceedings of SIGIR 1992, pp. 318--329.
• 6: Dey, A.K., Hamid, R., Beckmann, C., Li, I. and Hsu, D. a CAPpella: Programming by Demonstrations of Context-Aware Applications. Proceedings of CHI 2004, pp. 33--40.
• 7: Fails, J.A., Olsen Jr., D.R. Interactive Machine Learning. Proceedings of IUI 2003, pp. 39--45.
• 8: Fogarty, J., Tan. D., Kapoor, A. and Winder, S. CueFlik: Interactive Concept Learning in Image Search. Proceedings of CHI 2008, pp. 29--38.
• 9: Frazier, M., Goldman, S., Mishra, N. and Pitt, L. Learning from a Consistently Ignorant Teacher. Journal of Computing Systems Science 52, 3 (1996), 472--492.
• 10: Gorkani, M.M. and Picard, R.W. Texture Orientation for Sorting Photos 'At a Glance'. Proceedings of ICPR 1994, pp. 459--464.
• 11: Guo, Y. and Schuurmans, D. Discriminative Batch Mode Active Learning. Proceedings of NIPS 2007.
• 12: Gurevich, N., Markovitch, S. and Rivlin, E. Active Learning with Near Misses. Proceedings of AAAI 2006, pp. 362--367.
• 13: Hartmann, B., Abdulla, L., Mittal, M. and Klemmer, S.R. Authoring Sensor-Based Interactions by Demonstration with Direct Manipulation and Pattern Recognition. Proceedings of CHI 2007, pp. 145--154.
• 14: Hoffman, R., Amershi, S., Patel, K., Wu, F., Fogarty, J. and Weld, D.S. Amplifying Community Content Creation with Mixed-Initiative Information Extraction. Proceedings of CHI 2009, pp. 1849--1858.
• 15: Horvitz, E. Principles of Mixed-Initiative User Interfaces. Proceedings of CHI 1999, pp. 159--166.
• 16: Krause, A., Singh, A. and Guestrin, C. Near-optimal Sensor Placements in Gaussian Processes: Theory, Efficient Algorithms and Empirical Studies. Journal of Machine Learning Research 9, (2008), 235--284.
• 17: Kristjannson, T., Culotta, A., Viola, P. and McCallum, A. Interactive Information Extraction with Constrained Conditional Random Fields. Proceedings of AAAI 2004, pp. 412--418.
• 18: Muslea, I., Minton, S. and Knoblock, C.A. Active Learning with Multiple Views. Journal of AI Research 27, (2006), 203--233.
• 19: Ritter, A. and Basu, S. Learning to Generalize for Complex Selection Tasks. Proceedings of IUI 2009, pp. 167--176.
• 20: Schettini, R., Ciocca, G., Valsasna, A., Brambilla, C. and De Ponti, M. A Hierarchical Classification Strategy for Digital Documents. Pattern Recognition 35, 8 (2002), 1759--1769.
• 21: Shilman, M., Tan, D.S. and Simar, P. CueTIP: A Mixed-Initiative Interface for Correcting Handwriting Errors. Proceedings of UIST 2006, pp. 323--332.
• 22: Tenenbaum, J.B., de Silva, V. and Langford, J.C. A Global Geometric Framework for Nonlinear Dimensionality Reduction. Science 290, 5500 (2000), 2319--2323.
• 23: Terry, M. and Mynatt, E.D. Side Views: Persistent, On-Demand Previews for Open-Ended Tasks. Proceedings of UIST 2002, pp. 71--80.
• 24: Vailaya, A., Figueiredo, M., Jain, A. and Zhang, H.J. Content-Based Hierarchical Classification of Vacation Images. Proceedings of ICMCS 1999, pp. 518--523.
• 25: Zhu, X. Semi-supervised learning literature survey. Technical Report 1530, Department of Computer Sciences, University of Wisconsin, Madison, 2005.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Interaction styles (e.g., commands, menus, forms, direct manipulation)

Additional Classification:
  H. Information Systems
  H.1 MODELS AND PRINCIPLES
      H.1.2 User/Machine Systems
          Subjects: Human factors

General Terms:
Design, Human Factors

Keywords:
end-user interactive concept learning