From active towards InterActive learning

Subscribe (Full Service)


	Search: The ACM Digital Library The Guide

	Feedback

From active towards InterActive learning: using consideration information to improve labeling correctness

Full text

Pdf (270 KB)

Source

International Conference on Knowledge Discovery and Data Mining archive
Proceedings of the ACM SIGKDD Workshop on Human Computation table of contents

Paris, France

SESSION: Human computation in practice table of contents

Pages 40-43

Year of Publication: 2009

ISBN:978-1-60558-672-4

Authors

Abraham Bernstein	University of Zurich, Zürich, Switzerland
Jiwen Li	University of Zurich, Zürich, Switzerland

Sponsors

Microsoft Research : Microsoft Research
: Carnegie Mellon

Publisher

ACM New York, NY, USA

Bibliometrics

Downloads (6 Weeks): 7, Downloads (12 Months): 7, Citation Count: 0

Additional Information:

abstract references index terms

Tools and Actions:	Request Permissions Review this Article Save this Article to a Binder Display Formats: BibTeX EndNote ACM Ref

DOI Bookmark:	Use this link to bookmark this Article: http://doi.acm.org/10.1145/1600150.1600165 What is a DOI?

ABSTRACT

Active learning methods have been proposed to reduce the labeling effort of human experts: based on the initially available labeled instances and information about the unlabeled data those algorithms choose only the most informative instances for labeling. They have been shown to significantly reduce the size of the required labeled dataset to generate a precise model [17]. However, active learning framework assumes "perfect" labelers, which is not true in practice (e.g., [22, 23]). In particular, an empirical study for hand-written digit recognition [5] has shown that active learning works poorly when a human labeler is used. Thus, as active learning enters the realm of practical applications, it will need to confront the practicalities and inaccuracies of human expert decision-making. Specifically, active learning approaches will have to deal with the problem that human experts are likely to make mistakes when labeling the selected instances.

REFERENCES

Note: OCR errors may be found in this Reference List extracted from the full text article. ACM has opted to expose the complete List rather than only correct and linked references.

	1	Aha, D., Kibler, D., and Albert, M. "Instance-based learning algorithms," Machine Learning (6:1) 1991, pp 37--66.
	2	Alba, W. J., and Hutchinson, J. W. "Effects of Context and Post-Category on Recall of Competing Brands," Journal of Consumer Research (13:3) 1987, pp 411--454.
	3	Anderson, J. R. The Architecture of Cognition Harvard University Press, Cambridge, MA, 1983.
	4	Baron, J. Thinking AND Deciding Cambridge University Press, 1998.
	5	Baum, E. B., and Lang, K. "Query learning can work poorly when a human oracle is used," International Joint Conference in Neural Networks (IJCNN'92), Beijing, China, 1992.
	6	Bernstein, A., and Li, J. "From Active Towards InterActive Learning: Using Consideration Information to Improve Labeling Correctness." University of Zurich, Department of Informatics, Dynamic and Distributed Information Systems Group Working Paper. www.ifi.uzh.ch/ddis/nc/publications.
	7	Dawid, A. P., and Skene, A. M. "Maximum likelihood estimation of observer error-rates using the EM algorithm." Applied Statistics 28, 1 (Sept. 1979), pp 20--28.
	8	Frank, E., and Witten, I. H. "Generating Accurate Rule Sets Without Global Optimization," The Fifteenth International Conference in Machine Learning (ICML), Morgan Kaufmann Publishers, Madison, WI, 1998, pp. 144--151.
	9	Hamilton, R. W. "Why Do People Suggest What They Do Not Want? Using Context Effects to Influence Others' Choices," Journal of Consumer Research (29) 2003, pp 492--506.
	10	Hauser, J. R., and Wernerfelt, B. "An evaluation Cost Model of Consideration Sets," Journal of Consumer Research (16:4) 1990, pp 393--408.
	11	Jolliffe, I. T. Principal Component Analysis Springer; 2nd edition, 2002.
	12	Kahneman, D., Slovic, P., and Tversky, A. Judgment under uncertainty: Heuristics and biases Cambridge University Press, 1982.
	13	Laird, J. E., Newell, A., and Rosenbloom, P. S. "SOAR: An architecture for general intelligence," Artificial Intelligence (33:1) 1987, pp 1--64.
	14	Lugosi, G. "Learning with an unreliable teacher." Pattern Recognition 25, 1 (Jan. 1992), pp 79--87.
	15	Miller, A. G. "The Magical Number Seven, Plus or Minus Two: Some Limits on Our Capacity for Processing In formation," The Psychological Review (63) 1956, pp 81--97.
	16	Roberts, J. H., and James, L. L. "Consideration: Review of Research and Prospects for Future Insights," Jounral of Marketing research (34:3) 1997, pp 406--410.
	17	Saar-Tsechansky, M., and Provost, F. "Active Sampling for Class Probability Estimation and Ranking," Machine Learning (54:2) 2004, pp 153--178.
	18	Sheng, S., Provost, F., and Ipeirotis, P. "Get another label? Improving data quality and data mining using multiple, noisy labelers." Proceeding of the 14^th ACM SIGKDD international conference on Knowledge discovery and data mining. 2008, pp 614--622.
	19	Silverman, B. W. "Some asymptotic properties of the probabilistic teacher." IEEE Transactions on Information Theory 26, 2 (Mar 1980), pp 246--249.
	20	Smyth, P. "Learning with probabilistic supervision." In Computational Learning Theory and Natural Learning Systems. Vol. III: Selecting Good Models, T. Petsche, Ed. MIT Press, Apr. 1995.
	21	Smyth, P. "Bounds on the mean classification error rate of multiple experts." Pattern Recognition Letters 17, 12 (May 1996)
	22	Smyth, P., Burl, M. C, Fayyad, U. M., and Perona, P. "Knowledge discovery in large image databases: Dealing with uncertainties in ground truth." In Knowledge Discovery in Databases: Papers from the 1994 AAAI Workshop (KDD-94) 1994, pp 109--120.
	23	Smyth, P., Burl, M. C., Fayyad, U. M., and Perona, P. "Inferring ground truth from subjective labeling of Venus images." In NIPS 1994, pp 1085--1092.