Query by committee

Subscribe (Full Service)


	Search: The ACM Digital Library The Guide

	Feedback
Take a look at the new version of this page: [ beta version ]. Tell us what you think.

Query by committee

Full text

Pdf (775 KB)

Source

Annual Workshop on Computational Learning Theory archive
Proceedings of the fifth annual workshop on Computational learning theory table of contents

Pittsburgh, Pennsylvania, United States

Pages: 287 - 294

Year of Publication: 1992

ISBN:0-89791-497-X

Authors

H. S. Seung	Racah Institute of Physics and Center for Neural Computation, Hebrew University, Jerusalem 91904, Israel
M. Opper	Institut für Theoretische Physik, Justus-Liebig-Universität Giessen, D-6300 Giessen, Germany
H. Sompolinsky	Racah Institute of Physics and Center for Neural Computation, Hebrew University, Jerusalem 91904, Israel

Sponsors

SIGACT: ACM Special Interest Group on Algorithms and Computation Theory
SIGART: ACM Special Interest Group on Artificial Intelligence

Publisher

ACM New York, NY, USA

Bibliometrics

Downloads (6 Weeks): 13, Downloads (12 Months): 86, Citation Count: 101

Additional Information:

abstract references cited by index terms collaborative colleagues

Tools and Actions:	Review this Article Save this Article to a Binder Display Formats: BibTeX EndNote ACM Ref

DOI Bookmark:	Use this link to bookmark this Article: http://doi.acm.org/10.1145/130385.130417 What is a DOI?

ABSTRACT

We propose an algorithm called query by commitee, in which a committee of students is trained on the same data set. The next query is chosen according to the principle of maximal disagreement. The algorithm is studied for two toy models: the high-low game and perceptron learning of another perceptron. As the number of queries goes to infinity, the committee algorithm yields asymptotically finite information gain. This leads to generalization error that decreases exponentially with the number of examples. This in marked contrast to learning from randomly chosen inputs, for which the information gain approaches zero and the generalization error decreases with a relatively slow inverse power law. We suggest that asymptotically finite information gain may be an important characteristic of good query algorithms.

REFERENCES

Note: OCR errors may be found in this Reference List extracted from the full text article. ACM has opted to expose the complete List rather than only correct and linked references.

	Bau91	E. Baum. Neural net algorithms that learn in polynomial time from examples and queries. IEEE Trans. in Neural Networks, 2:5-19, 1991.
	Fed72	V. V. Fedorov. Theory of Optimal Experiments. Academic Press, New York, 1972.
	GD89	E. Gardner and B. Derrida. Three unfinished works on the optimal storage capacity of networks. J. Phys., A22:1983-1994, 1989.
	GT90	G. GySrgyi and N. Tishby. Statistical theory oxc learn{rig a rule. In W. I~. rI)heumann and R. KSberle, editors, Neural Networks and Spin Glasses, pages 3-36, Singapore, 1990. World Scientific.
	HKS91	David Haussler , Michael Kearns , Robert Schapire, Bounds on the sample complexity of Bayesian learning using information theory and the VC dimension, Proceedings of the fourth annual workshop on Computational learning theory, p.61-74, August 05-07, 1991, Santa Cruz, California, United States
	KR90	W. Kinzel and P. Ruj~n. Improving a network generalization ability by selecting examples. Europhys. Lett., 13:473-477, 1990.
	MP88	M. L. Minsky and S. Papert. Percep~rons. MIT Press, Cambridge, expanded edition, 1988.
	OH91	M. Opper and D. Haussler. Generalization performance of bayes optimal classification algorithm for learning a perceptron. Phys. Rev. Left., 66:2677-2680, 1991.
	SST92	H. S. Seung, H. Sompolinsky, and N. Tishby. Statistical mechanics of learning from examples. Phys. Rev., A45:6056-6091, 1992.
	TLS89	N. Tishby, E. Levin, and S. Solla. Consistent inference of probabilities in layered networks: Predictions and generalization. In Proc. Int. Joint Conf. on Neural Networks, volume 2, pages 403-409, Washington, DC, 1989. IEEE.
	WR92	T.L.H. Watkin and A. Rau. Selecting examples for perceptrons. J. Phys., A25:113-121, 1992.

CITED BY 101

	Christopher J. Merz , Michael J. Pazzani, A Principal Components Approach to Combining Regression Estimates, Machine Learning, v.36 n.1-2, p.9-32, July-August 1999
	M. Hasenjäger , H. Ritter, Active learning in neural networks, New learning paradigms in soft computing, Physica-Verlag GmbH, Heidelberg, Germany, 2002
	Amit Singhal , Mandar Mitra , Chris Buckley, Learning routing queries in a query zone, ACM SIGIR Forum, v.31 n.SI, p.25-32
	David D. Lewis , William A. Gale, A sequential algorithm for training text classifiers, Proceedings of the 17th annual international ACM SIGIR conference on Research and development in information retrieval, p.3-12, July 03-06, 1994, Dublin, Ireland
	Sunita Sarawagi , Anuradha Bhamidipaty, Interactive deduplication using active learning, Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, July 23-26, 2002, Edmonton, Alberta, Canada
	David Helmbold , Sandra Panizza, Some label efficient learning results, Proceedings of the tenth annual conference on Computational learning theory, p.218-230, July 06-09, 1997, Nashville, Tennessee, United States
	Vijay S. Iyengar , Chidanand Apte , Tong Zhang, Active learning using adaptive resampling, Proceedings of the sixth ACM SIGKDD international conference on Knowledge discovery and data mining, p.91-98, August 20-23, 2000, Boston, Massachusetts, United States
	Simon Tong , Edward Chang, Support vector machine active learning for image retrieval, Proceedings of the ninth ACM international conference on Multimedia, September 30-October 05, 2001, Ottawa, Canada
	A. P. Engelbrecht , R. Brits, Supervised Training Using an Unsupervised Approach to Active Learning, Neural Processing Letters, v.15 n.3, p.247-260, June 2002
	Simon Tong , Daphne Koller, Support vector machine active learning with applications to text classification, The Journal of Machine Learning Research, 2, p.45-66, 3/1/2002
	Shai Fine , Ran Gilad-Bachrach , Eli Shamir, Query by committee, linear separation and random walks, Theoretical Computer Science, v.284 n.1, p.25-51, 6 July 2002
	Sean P. Engelson , Ido Dagan, Minimizing manual annotation cost in supervised training from corpora, Proceedings of the 34th annual meeting on Association for Computational Linguistics, p.319-326, June 24-27, 1996, Santa Cruz, California
	Michael Lindenbaum , Shaul Markovitch , Dmitry Rusakov, Selective Sampling for Nearest Neighbor Classifiers, Machine Learning, v.54 n.2, p.125-152, February 2004
	Maytal Saar-Tsechansky , Foster Provost, Active Sampling for Class Probability Estimation and Ranking, Machine Learning, v.54 n.2, p.153-178, February 2004
	Jens Kaae Christensen , Kasper Lamberth , Morten Nielsen , Claus Lundegaard , Peder Worning , Sanne Lise Lauemøller , Søren Buus , Søren Brunak , Ole Lund, Selecting informative data for developing peptide-MHC binding predictors using a query by committee approach, Neural Computation, v.15 n.12, p.2931-2942, December 2003
	Atsushi Fujii , Takenobu Tokunaga , Kentaro Inui , Hozumi Tanaka, Selective sampling for example-based word sense disambiguation, Computational Linguistics, v.24 n.4, p.573-597, December 1998
	Dilek Hakkani-Tür , Giuseppe Riccardi , Gokhan Tur, An active approach to spoken language processing, ACM Transactions on Speech and Language Processing (TSLP), v.3 n.3, p.1-31, October 2006
	Rong Jin , Luo Si, A Bayesian approach toward active learning for collaborative filtering, Proceedings of the 20th conference on Uncertainty in artificial intelligence, p.278-285, July 07-11, 2004, Banff, Canada
	Prem Melville , Raymond J. Mooney, Diverse ensembles for active learning, Proceedings of the twenty-first international conference on Machine learning, p.74, July 04-08, 2004, Banff, Alberta, Canada
	Haym Hirsh , Nina Mishra , Leonard Pitt, Version spaces and the consistency problem, Artificial Intelligence, v.156 n.2, p.115-138, July 2004
	Rong Jin , Joyce Y. Chai , Luo Si, Effective automatic image annotation via a coherent language model and active learning, Proceedings of the 12th annual ACM international conference on Multimedia, October 10-16, 2004, New York, NY, USA
	Sanjoy Dasgupta , Wee Sun Lee , Philip M. Long, A Theoretical Analysis of Query Selection for Collaborative Filtering, Machine Learning, v.51 n.3, p.283-298, June 2003
	Klaus Brinker, Active learning of label ranking functions, Proceedings of the twenty-first international conference on Machine learning, p.17, July 04-08, 2004, Banff, Alberta, Canada
	Radu Florian , John C. Henderson , Grace Ngai, Coaxing confidences from an old friend: probabilistic classifications from transformation rule lists, Proceedings of the 2000 Joint SIGDAT conference on Empirical methods in natural language processing and very large corpora: held in conjunction with the 38th Annual Meeting of the Association for Computational Linguistics, p.26-34, October 07-08, 2000, Hong Kong
	Li Zhang , ShiXia Liu , Yue Pan , LiPing Yang, InfoAnalyzer: a computer-aided tool for building enterprise taxonomies, Proceedings of the thirteenth ACM international conference on Information and knowledge management, November 08-13, 2004, Washington, D.C., USA
	Grace Ngai , David Yarowsky, Rule writing or annotation: cost-efficient resource usage for base noun phrase chunking, Proceedings of the 38th Annual Meeting on Association for Computational Linguistics, p.117-125, October 03-06, 2000, Hong Kong
	Zhi-Hua Zhou , Ming Li, Tri-Training: Exploiting Unlabeled Data Using Three Classifiers, IEEE Transactions on Knowledge and Data Engineering, v.17 n.11, p.1529-1541, November 2005
	Xingquan Zhu , Xindong Wu, Cost-Constrained Data Acquisition for Intelligent Data Preparation, IEEE Transactions on Knowledge and Data Engineering, v.17 n.11, p.1542-1556, November 2005
	Brigham Anderson , Andrew Moore, Active learning for Hidden Markov Models: objective functions and algorithms, Proceedings of the 22nd international conference on Machine learning, p.9-16, August 07-11, 2005, Bonn, Germany
	Yuval Krymolowski, Distinguishing easy and hard instances, proceeding of the 6th conference on Natural language learning, p.1-6, August 31, 2002
	Yoram Baram , Ran El-Yaniv , Kobi Luz, Online Choice of Active Learning Algorithms, The Journal of Machine Learning Research, 5, p.255-291, 12/1/2004
	Arnd Christian König , Eric Brill, Reducing the human overhead in text categorization, Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, August 20-23, 2006, Philadelphia, PA, USA
	Zhi-Hua Zhou , Ke-Jia Chen , Hong-Bin Dai, Enhancing relevance feedback in image retrieval using unlabeled data, ACM Transactions on Information Systems (TOIS), v.24 n.2, p.219-244, April 2006
	Li Zhang , Tao Li , Shixia Liu , Yue Pan, An integrated system for building enterprise taxonomies, Information Retrieval, v.10 n.4-5, p.365-391, October 2007
	Wang Ye , Huang Shang-Teng, Reducing the number of sub-classifiers for pairwise multi-category support vector machines, Pattern Recognition Letters, v.28 n.15, p.2088-2093, November, 2007
	Shai Fine , Yishay Mansour, Active sampling for multiple output identification, Machine Learning, v.69 n.2-3, p.213-228, December 2007
	Andrew I. Schein , Lyle H. Ungar, Active learning for logistic regression: an evaluation, Machine Learning, v.68 n.3, p.235-265, October 2007
	Francois Barbançon , Daniel P. Miranker, SPHINX: Schema integration by example, Journal of Intelligent Information Systems, v.29 n.2, p.145-184, October 2007
	Mingxi Wu , Christopher Jermaine, Outlier detection by sampling with accuracy guarantees, Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, August 20-23, 2006, Philadelphia, PA, USA
	Sriharsha Veeramachaneni , Emanuele Olivetti , Paolo Avesani, Active sampling for detecting irrelevant features, Proceedings of the 23rd international conference on Machine learning, p.961-968, June 25-29, 2006, Pittsburgh, Pennsylvania
	Steven C. H. Hoi , Rong Jin , Jianke Zhu , Michael R. Lyu, Batch mode active learning and its application to medical image classification, Proceedings of the 23rd international conference on Machine learning, p.417-424, June 25-29, 2006, Pittsburgh, Pennsylvania
	Naoki Abe , Bianca Zadrozny , John Langford, Outlier detection by active learning, Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, August 20-23, 2006, Philadelphia, PA, USA
	Steven C. H. Hoi , Rong Jin , Michael R. Lyu, Large-scale text categorization by batch mode active learning, Proceedings of the 15th international conference on World Wide Web, May 23-26, 2006, Edinburgh, Scotland
	Hinrich Schütze , Emre Velipasaoglu , Jan O. Pedersen, Performance thresholding in practical text classification, Proceedings of the 15th ACM international conference on Information and knowledge management, November 06-11, 2006, Arlington, Virginia, USA
	Bishan Yang , Jian-Tao Sun , Tengjiao Wang , Zheng Chen, Effective multi-label active learning for text classification, Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining, June 28-July 01, 2009, Paris, France
	Robbie Haertel , Eric Ringger , Kevin Seppi , James Carroll , Peter McClanahan, Assessing the costs of sampling methods in active learning for annotation, Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics on Human Language Technologies: Short Papers, June 16-17, 2008, Columbus, Ohio
	Ran Avnimelech , Nathan Intrator, Boosted mixture of experts: an ensemble learning scheme, Neural Computation, v.11 n.2, p.483-497, Feb. 15, 1999
	Josh Bongard, Action-selection and crossover strategies for self-modeling machines, Proceedings of the 9th annual conference on Genetic and evolutionary computation, July 07-11, 2007, London, England
	Zuobing Xu , Ram Akella, A bayesian logistic regression model for active relevance feedback, Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval, July 20-24, 2008, Singapore, Singapore
	Josh Bongard, Exploiting multiple robots to accelerate self-modeling, Proceedings of the 9th annual conference on Genetic and evolutionary computation, July 07-11, 2007, London, England
	Josh Bongard , Hod Lipson, Active Coevolutionary Learning of Deterministic Finite Automata, The Journal of Machine Learning Research, 6, p.1651-1678, 12/1/2005
	Andreas Vlachos, A stopping criterion for active learning, Computer Speech and Language, v.22 n.3, p.295-312, July, 2008
	Yoav Freund , H. Sebastian Seung , Eli Shamir , Naftali Tishby, Selective Sampling Using the Query by Committee Algorithm, Machine Learning, v.28 n.2-3, p.133-168, Aug./Sept. 1997
	A. P. Engelbrecht, Sensitivity Analysis for Selective Learning by Feedforward Neural Networks, Fundamenta Informaticae, v.46 n.3, p.219-252, August 2001
	Andries P. Engelbrecht, Sensitivity Analysis for Selective Learning by Feedforward Neural Networks, Fundamenta Informaticae, v.45 n.4, p.295-328, December 2001
	Martina Hasenjäger , Helge Ritter, Active Learning with Local Models, Neural Processing Letters, v.7 n.2, p.107-117, April 1998
	Huan Liu , Hiroshi Motoda, On Issues of Instance Selection, Data Mining and Knowledge Discovery, v.6 n.2, p.115-130, April 2002
	Aleksander Kołcz , Joshua Alspector, Asymmetric Missing-data Problems: Overcoming the Lack of Negative Data in Preference Ranking, Information Retrieval, v.5 n.1, p.5-40, January 2002
	Zhenyu Lu , Anand I. Rughani , Bruce I. Tranmer , Josh Bongard, Informative sampling for large unbalanced data sets, Proceedings of the 2008 GECCO conference companion on Genetic and evolutionary computation, July 12-16, 2008, Atlanta, GA, USA
	Stephen Purpura , Claire Cardie , Jesse Simons, Active learning for e-rulemaking: public comment categorization, Proceedings of the 2008 international conference on Digital government research, May 18-21, 2008, Montreal, Canada
	Shihao Ji , Balaji Krishnapuram , Lawrence Carin, Variational Bayes for Continuous Hidden Markov Models and Its Application to Active Learning, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.28 n.4, p.522, April 2006
	György J. Simon , Vipin Kumar , Zhi-Li Zhang, Semi-supervised approach to rapid and reliable labeling of large data sets, Proceeding of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining, August 24-27, 2008, Las Vegas, Nevada, USA
	Charles X. Ling , Jun Du, Active learning with direct query construction, Proceeding of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining, August 24-27, 2008, Las Vegas, Nevada, USA
	Stéphane Ayache , Georges Quénot, Evaluation of active learning strategies for video indexing, Image Communication, v.22 n.7-8, p.692-704, August, 2007
	Foster Provost , Prem Melville , Maytal Saar-Tsechansky, Data acquisition and cost-effective predictive modeling: targeting offers for electronic commerce, Proceedings of the ninth international conference on Electronic commerce, August 19-22, 2007, Minneapolis, MN, USA
	Takafumi Kanamori, Pool-based active learning with optimal sampling distribution and its information geometrical interpretation, Neurocomputing, v.71 n.1-3, p.353-362, December, 2007
	Sonia Chernova , Manuela Veloso, Multi-thresholded approach to demonstration selection for interactive robot learning, Proceedings of the 3rd ACM/IEEE international conference on Human robot interaction, March 12-15, 2008, Amsterdam, The Netherlands
	Harris Drucker , Corinna Cortes , L. D. Jackel , Yann LeCun , Vladimir Vapnik, Boosting and other ensemble methods, Neural Computation, v.6 n.6, p.1289-1301, November 1994
	Jason Baldridge , Miles Osborne, Active learning and logarithmic opinion pools for hpsg parse selection, Natural Language Engineering, v.14 n.2, p.191-222, April 2008
	Constantinos Constantinopoulos , Aristidis Likas, Semi-supervised and active learning with the probabilistic RBF classifier, Neurocomputing, v.71 n.13-15, p.2489-2498, August, 2008
	Shucheng Huang , Yisheng Dong, An active learning system for mining time-changing data streams, Intelligent Data Analysis, v.11 n.4, p.401-419, September 2007
	Abhay S. Harpale , Yiming Yang, Personalized active learning for collaborative filtering, Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval, July 20-24, 2008, Singapore, Singapore
	Ricardo B. C. Prudêncio , Teresa B. Ludermir, Selective generation of training examples in active meta-learning, International Journal of Hybrid Intelligent Systems, v.5 n.2, p.59-70, April 2008
	Matthias W. Seeger, Bayesian Inference and Optimal Design for the Sparse Linear Model, The Journal of Machine Learning Research, 9, p.759-813, 6/1/2008
	Falk-Florian Henrich , Klaus Obermayer, Active Learning by Spherical Subdivision, The Journal of Machine Learning Research, 9, p.105-130, 6/1/2008
	Shenghuo Zhu , Tao Li , Zhiyuan Chen , Dingding Wang , Yihong Gong, Dynamic active probing of helpdesk databases, Proceedings of the VLDB Endowment, v.1 n.1, August 2008
	Zhi-Hua Zhou , Ming Li, Semisupervised Regression with Cotraining-Style Algorithms, IEEE Transactions on Knowledge and Data Engineering, v.19 n.11, p.1479-1493, November 2007
	Feilong Chen , Rong Jin, Active algorithm selection, Proceedings of the 22nd national conference on Artificial intelligence, p.534-539, July 22-26, 2007, Vancouver, British Columbia, Canada
	Nitesh V. Chawla , Kevin W. Bowyer, Actively exploring creation of face space(s) for improved face recognition, Proceedings of the 22nd national conference on Artificial intelligence, p.809-814, July 22-26, 2007, Vancouver, British Columbia, Canada
	Gang Xiao , Finnegan Southey , Robert C. Holte , Dana Wilkinson, Software testing by active learning for commercial games, Proceedings of the 20th national conference on Artificial intelligence, p.898-903, July 09-13, 2005, Pittsburgh, Pennsylvania
	Krishnanand N. Kaipa , Josh C. Bongard , Andrew N. Meltzoff, Combined structure and motion extraction from visual data using evolutionary active learning, Proceedings of the 11th Annual conference on Genetic and evolutionary computation, July 08-12, 2009, Montreal, Québec, Canada
	Charles Jackson , Robert F. Murphy , Jelena Kovačcevic, Intelligent acquisition and learning of fluorescence microscope data models, IEEE Transactions on Image Processing, v.18 n.9, p.2071-2084, September 2009
	Dan Goldwasser , Dan Roth, Active sample selection for named entity transliteration, Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics on Human Language Technologies: Short Papers, June 16-17, 2008, Columbus, Ohio
	Cynthia A. Thompson , Raymond J. Mooney, Acquiring word-meaning mappings for natural language interfaces, Journal of Artificial Intelligence Research, v.18 n.1, p.1-44, January 2003
	Ion Muslea , Steven Minton , Craig A. Knoblock, Active learning with multiple views, Journal of Artificial Intelligence Research, v.27 n.1, p.203-233, September 2006
	Johannes Fürnkranz, Integrative Windowing, Journal of Artificial Intelligence Research, v.8 n.1, p.129-164, January 1998
	Weiming Hu , Wei Hu , Nianhua Xie , Steve Maybank, Unsupervised active learning based on hierarchical graph-theoretic clustering, IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics, v.39 n.5, p.1147-1161, October 2009
	Barnabás Póczos , András Lőrincz, Identification of Recurrent Neural Networks by Bayesian Interrogation Techniques, The Journal of Machine Learning Research, 10, p.515-554, June 2009
	Sanjoy Dasgupta , Adam Tauman Kalai , Claire Monteleoni, Analysis of Perceptron-Based Active Learning, The Journal of Machine Learning Research, 10, p.281-299, June 2009
	Ion Muslea , Steven N. Minton , Craig A. Knoblock, Active learning with strong and weak views: a case study on wrapper induction, Proceedings of the 18th international joint conference on Artificial intelligence, p.415-420, August 09-15, 2003, Acapulco, Mexico
	Maytal Saar-Tsechansky , Foster Provost, Active learning for class probability estimation and ranking, Proceedings of the 17th international joint conference on Artificial intelligence, p.911-917, August 04-10, 2001, Seattle, WA, USA
	Markus Becker , Miles Osborne, A two-stage method for active learning of statistical grammars, Proceedings of the 19th international joint conference on Artificial intelligence, p.991-996, July 30-August 05, 2005, Edinburgh, Scotland
	Alexander Liu , Goo Jun , Joydeep Ghosh, A self-training approach to cost sensitive uncertainty sampling, Machine Learning, v.76 n.2-3, p.257-270, September 2009
	Erik Boiy , Marie-Francine Moens, A machine learning approach to sentiment analysis in multilingual Web texts, Information Retrieval, v.12 n.5, p.526-558, October 2009
	Zhenyu Lu , Josh Bongard, Exploiting multiple classifier types with active learning, Proceedings of the 11th Annual conference on Genetic and evolutionary computation, July 08-12, 2009, Montreal, Québec, Canada
	Josh C. Bongard, Accelerating self-modeling in cooperative robot teams, IEEE Transactions on Evolutionary Computation, v.13 n.2, p.321-332, April 2009
	Tina Yu , Dave Wilkinson, Coevolution of simulator proxies and sampling strategies for petroleum reservoir modeling, Proceedings of the Eleventh conference on Congress on Evolutionary Computation, p.2677-2684, May 18-21, 2009, Trondheim, Norway
	Katrin Tomanek , Joachim Wermter , Udo Hahn, Efficient annotation with the Jena ANnotation Environment (JANE), Proceedings of the Linguistic Annotation Workshop, p.9-16, June 28-29, 2007, Prague, Czech Republic
	Dan Zhang , Fei Wang , Zhenwei Shi , Changshui Zhang, Interactive localized content based image retrieval with multiple-instance active learning, Pattern Recognition, v.43 n.2, p.478-484, February, 2010
	Eric Ringger , Peter McClanahan , Robbie Haertel , George Busby , Marc Carmen , James Carroll , Kevin Seppi , Deryle Lonsdale, Active learning for part-of-speech tagging: accelerating corpus annotation, Proceedings of the Linguistic Annotation Workshop, p.101-108, June 28-29, 2007, Prague, Czech Republic
	Burr Settles , Mark Craven, An analysis of active learning strategies for sequence labeling tasks, Proceedings of the Conference on Empirical Methods in Natural Language Processing, October 25-27, 2008, Honolulu, Hawaii