EnsembleMatrix: interactive visualization to support machine learning with multiple classifiers

ABSTRACT

Machine learning is an increasingly used computational tool within human-computer interaction research. While most researchers currently utilize an iterative approach to refining classifier models and performance, we propose that ensemble classification techniques may be a viable and even preferable alternative. In ensemble learning, algorithms combine multiple classifiers to build one that is superior to its components. In this paper, we present EnsembleMatrix, an interactive visualization system that presents a graphical view of confusion matrices to help users understand relative merits of various classifiers. EnsembleMatrix allows users to directly interact with the visualizations in order to explore and build combination models. We evaluate the efficacy of the system and the approach in a user study. Results show that users are able to quickly combine multiple classifiers operating on multiple feature sets to produce an ensemble classifier with accuracy that approaches best-reported performance classifying images in the CalTech-101 dataset.

REFERENCES

• 1: Mihael Ankerst , Christian Elsen , Martin Ester , Hans-Peter Kriegel, Visual classification: an interactive approach to decision tree construction, Proceedings of the fifth ACM SIGKDD international conference on Knowledge discovery and data mining, p.392-396, August 15-18, 1999, San Diego, California, United States [doi>10.1145/312129.312298]
• 2: Becker, B., Kohavi, R. and Sommerfield, D. Visualizing the Simple Bayesian Classifier. {ed.} Fayyad, U., Grinstein, G. and Wierse, A. (2001), 237--249.
• 3: Jacques Bertin, Semiology of graphics, University of Wisconsin Press, 1983
• 4: Anna Bosch , Andrew Zisserman , Xavier Munoz, Representing shape with a spatial pyramid kernel, Proceedings of the 6th ACM international conference on Image and video retrieval, p.401-408, July 09-11, 2007, Amsterdam, The Netherlands [doi>10.1145/1282280.1282340]
• 5: Leo Breiman, Bagging predictors, Machine Learning, v.24 n.2, p.123-140, Aug. 1996 [doi>10.1023/A:1018054314350]
• 6: Doina Caragea , Dianne Cook , Vasant G. Honavar, Gaining insights into support vector machine pattern classifiers using projection-based tour methods, Proceedings of the seventh ACM SIGKDD international conference on Knowledge discovery and data mining, p.251-256, August 26-29, 2001, San Francisco, California [doi>10.1145/502512.502547]
• 7: Dai, J. and Cheng, J. HMMEditor: a visual editing tool for profile hidden Markov models. BMC Genomics 2008, 9 (2008).
• 8: Chris Drummond , Robert C. Holte, Cost curves: An improved method for visualizing classifier performance, Machine Learning, v.65 n.1, p.95-130, October 2006 [doi>10.1007/s10994-006-8199-5]
• 9: Theodoros Evgeniou , Massimiliano Pontil , André Elisseeff, Leave One Out Error, Stability, and Generalization of Voting Combinations of Classifiers, Machine Learning, v.55 n.1, p.71-97, April 2004 [doi>10.1023/B:MACH.0000019805.88351.60]
• 10: Jerry Alan Fails , Dan R. Olsen, Jr., Interactive machine learning, Proceedings of the 8th international conference on Intelligent user interfaces, January 12-15, 2003, Miami, Florida, USA [doi>10.1145/604045.604056]
• 11: Li Fei-Fei , Rob Fergus , Pietro Perona, Learning Generative Visual Models from Few Training Examples: An Incremental Bayesian Approach Tested on 101 Object Categories, Proceedings of the 2004 Conference on Computer Vision and Pattern Recognition Workshop (CVPRW'04) Volume 12, p.178, June 27-July 02, 2004
• 12: James Fogarty , Andrew J. Ko , Htet Htet Aung , Elspeth Golden , Karen P. Tang , Scott E. Hudson, Examining task engagement in sensor-based statistical models of human interruptibility, Proceedings of the SIGCHI conference on Human factors in computing systems, April 02-07, 2005, Portland, Oregon, USA [doi>10.1145/1054972.1055018]
• 13: Frank, E. and Hall, M. Visualizing class probability estimators. Lecture Notes in Artificial Intelligence 2838, Springer (2003), 168--179.
• 14: Garner, S.R. WEKA: The Waikato Environment for Knowledge Analysis. Proc. New Zealand Computer Science Research Students Conference (1995), 57--64.
• 15: Kristen Grauman , Trevor Darrell, The Pyramid Match Kernel: Discriminative Classification with Sets of Image Features, Proceedings of the Tenth IEEE International Conference on Computer Vision, p.1458-1465, October 17-20, 2005 [doi>10.1109/ICCV.2005.239]
• 16: David Grimes , Desney S. Tan , Scott E. Hudson , Pradeep Shenoy , Rajesh P.N. Rao, Feasibility and pragmatics of classifying working memory load with an electroencephalograph, Proceeding of the twenty-sixth annual SIGCHI conference on Human factors in computing systems, April 05-10, 2008, Florence, Italy [doi>10.1145/1357054.1357187]
• 17: Nathalie Henry , Jean-Daniel Fekete, MatrixExplorer: a Dual-Representation System to Explore Social Networks, IEEE Transactions on Visualization and Computer Graphics, v.12 n.5, p.677-684, September 2006 [doi>10.1109/TVCG.2006.160]
• 18: Kapoor, A., Grauman, K., Urtasun, R. and Darrell, T. Active Learning with Gaussian Processes for Object Categorization. Proc. ICCV 2007, (2007), 1--8.
• 19: Ashish Kapoor , Eric Horvitz, Experience sampling for building predictive user models: a comparative study, Proceeding of the twenty-sixth annual SIGCHI conference on Human factors in computing systems, April 05-10, 2008, Florence, Italy [doi>10.1145/1357054.1357159]
• 20: Bohyoung Kim , Bongshin Lee , Jinwook Seo, Visualizing set concordance with permutation matrices and fan diagrams, Interacting with Computers, v.19 n.5-6, p.630-643, December, 2007 [doi>10.1016/j.intcom.2007.05.004]
• 21: Josef Kittler , Mohamad Hatef , Robert P. W. Duin , Jiri Matas, On Combining Classifiers, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.20 n.3, p.226-239, March 1998 [doi>10.1109/34.667881]
• 22: Svetlana Lazebnik , Cordelia Schmid , Jean Ponce, Beyond Bags of Features: Spatial Pyramid Matching for Recognizing Natural Scene Categories, Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, p.2169-2178, June 17-22, 2006 [doi>10.1109/CVPR.2006.68]
• 23: Mäkinen, E. and Siirtola, H. The Barycenter Heuristic and the Reorderable Matrix. Informatica (Slovenia), 29, 3 (2005), 357--364.
• 24: Mark Meyer , Alan Barr , Haeyoung Lee , Mathieu Desbrun, Generalized barycentric coordinates on irregular polygons, Journal of Graphics Tools, v.7 n.1, p.13-22, November 2002
• 25: Moustakis, V. Do People in HCI Use Machine Learning? HCI, 2, (1997), 95--98.
• 26: Nuria Oliver , Ashutosh Garg , Eric Horvitz, Layered representations for learning and inferring office activity from multiple sensory channels, Computer Vision and Image Understanding, v.96 n.2, p.163-180, November 2004 [doi>10.1016/j.cviu.2004.02.004]
• 27: Patel, K., Fogarty, J., Landay, J.A. and Harrison, B. Examining Difficulties Software Developers Encounter in the Adoption of Statistical Machine Learning. Proc. AAAI 2008, (2008), 1563--1566.
• 28: Rosalind W. Picard, Affective computing, MIT Press, Cambridge, MA, 1997
• 29: Penny Rheingans , Marie desJardins, Visualizing high-dimensional predicitive model quality, Proceedings of the conference on Visualization '00, p.493-496, October 2000, Salt Lake City, Utah, United States
• 30: T Scott Saponas , Desney S. Tan , Dan Morris , Ravin Balakrishnan, Demonstrating the feasibility of using forearm electromyography for muscle-computer interfaces, Proceeding of the twenty-sixth annual SIGCHI conference on Human factors in computing systems, April 05-10, 2008, Florence, Italy [doi>10.1145/1357054.1357138]
• 31: Schapire, R.E. The boosting approach to machine learning: An overview. Nonlinear Estimation and Classification. Springer (2003).
• 32: Gregor Stiglic , Matej Mertik , Vili Podgorelec , Peter Kokol, Using Visual Interpretation of Small Ensembles in Microarray Analysis, Proceedings of the 19th IEEE Symposium on Computer-Based Medical Systems, p.691-695, June 22-23, 2006 [doi>10.1109/CBMS.2006.169]
• 33: Urbanek, S. Exploring Statistical Forests. Proc. of the 2002 Joint Statistical Meeting, Springer (2002).
• 34: Varma, M. and Ray, D. Learning The Discriminative Power-Invariance Trade-Off. Proc. ICCV 2007, (2007), 1--8.
• 35: Wang, J., Yu, B. and Gasser, L. Classification Visualization with Shaded Similarity Matrix, Technical Report, GSLIS University of Illinois at Urbana-Champaign, (2002).
• 36: Malcolm Ware , Eibe Frank , Geoffrey Holmes , Mark Hall , Ian H. Witten, Interactive machine learning: letting users build classifiers, International Journal of Human-Computer Studies, v.56 n.3, p.281-292, September, 2002
• 37: Hao Zhang , Alexander C. Berg , Michael Maire , Jitendra Malik, SVM-KNN: Discriminative Nearest Neighbor Classification for Visual Category Recognition, Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, p.2126-2136, June 17-22, 2006 [doi>10.1109/CVPR.2006.301]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Graphical user interfaces (GUI)

Additional Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.6 Learning

General Terms:
Design

Keywords:
caltech-101, ensemble classifiers, interactive machine learning, object recognition, visualization