Image analogies

ABSTRACT

This paper describes a new framework for processing images by example, called “image analogies.” The framework involves two stages: a design phase, in which a pair of images, with one image purported to be a “filtered” version of the other, is presented as “training data”; and an application phase, in which the learned filter is applied to some new target image in order to create an “analogous” filtered result. Image analogies are based on a simple multi-scale autoregression, inspired primarily by recent results in texture synthesis. By choosing different types of source image pairs as input, the framework supports a wide variety of “image filter” effects, including traditional image filters, such as blurring or embossing; improved texture synthesis, in which some textures are synthesized with higher quality than by previous approaches; super-resolution, in which a higher-resolution image is inferred from a low-resolution source; texture transfer, in which images are “texturized” with some arbitrary source texture; artistic filters, in which various drawing and painting styles are synthesized based on scanned real-world examples; and texture-by-numbers, in which realistic scenes, composed of a variety of textures, are created using a simple painting interface.

REFERENCES

Note: OCR errors may be found in this Reference List extracted from the full text article. ACM has opted to expose the complete List rather than only correct and linked references.

	1	Sunil Arya , David M. Mount , Nathan S. Netanyahu , Ruth Silverman , Angela Y. Wu, An optimal algorithm for approximate nearest neighbor searching fixed dimensions, Journal of the ACM (JACM), v.45 n.6, p.891-923, Nov. 1998 [doi>10.1145/293347.293348]
	2	Michael Ashikhmin, Synthesizing natural textures, Proceedings of the 2001 symposium on Interactive 3D graphics, p.217-226, March 2001 [doi>10.1145/364338.364405]
	3	Michael F. Barnsley , Lyman P. Hurd, Fractal image compression, A. K. Peters, Ltd., Natick, MA, 1993
	4	Jeremy S. De Bonet, Multiresolution sampling procedure for analysis and synthesis of texture images, Proceedings of the 24th annual conference on Computer graphics and interactive techniques, p.361-368, August 1997 [doi>10.1145/258734.258882]
	5	Matthew Brand, Voice puppetry, Proceedings of the 26th annual conference on Computer graphics and interactive techniques, p.21-28, July 1999 [doi>10.1145/311535.311537]
	6	Matthew Brand , Aaron Hertzmann, Style machines, Proceedings of the 27th annual conference on Computer graphics and interactive techniques, p.183-192, July 2000 [doi>10.1145/344779.344865]
	7	Christoph Bregler , Michele Covell , Malcolm Slaney, Video Rewrite: driving visual speech with audio, Proceedings of the 24th annual conference on Computer graphics and interactive techniques, p.353-360, August 1997 [doi>10.1145/258734.258880]
	8	Ian Buck , Adam Finkelstein , Charles Jacobs , Allison Klein , David H. Salesin , Joshua Seims , Richard Szeliski , Kentaro Toyama, Performance-driven hand-drawn animation, Proceedings of the 1st international symposium on Non-photorealistic animation and rendering, p.101-108, June 05-07, 2000, Annecy, France [doi>10.1145/340916.340929]
	9	Kenneth R. Castleman, Digital image processing, Prentice Hall Press, Upper Saddle River, NJ, 1996
	10	Cassidy J. Curtis , Sean E. Anderson , Joshua E. Seims , Kurt W. Fleischer , David H. Salesin, Computer-generated watercolor, Proceedings of the 24th annual conference on Computer graphics and interactive techniques, p.421-430, August 1997 [doi>10.1145/258734.258896]
	11	Alexei A. Efros , Thomas K. Leung, Texture Synthesis by Non-Parametric Sampling, Proceedings of the International Conference on Computer Vision-Volume 2, p.1033, September 20-25, 1999
	12	Alexei A. Efros , William T. Freeman, Image quilting for texture synthesis and transfer, Proceedings of the 28th annual conference on Computer graphics and interactive techniques, p.341-346, August 2001 [doi>10.1145/383259.383296]
	13	Alex Eilhauer, Alice Pritikin, Dylan Weed, and Steven J. Gortler. Combining Textures and Pictures with Specialized Texture Synthesis, 2000. http://www.people.fas.harvard.edu/ ~pritikin/cs/graphics/.
	14	Chris Eliasmith. Dictionary of Philosophy of Mind. http://artsci.wustl.edu/ ~philos/MindDict/.
	15	T.G. Evans. A program for the solution of geometric analogy intelligence test questions. In M. Minsky, editor, Semantic Information Processing. MIT Press, 1968.
	16	James D. Foley , Andries van Dam , Steven K. Feiner , John F. Hughes, Computer graphics: principles and practice (2nd ed.), Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1990
	17	William T. Freeman , Egon C. Pasztor , Owen T. Carmichael, Learning Low-Level Vision, International Journal of Computer Vision, v.40 n.1, p.25-47, Oct. 2000 [doi>10.1023/A:1026501619075]
	18	William T. Freeman, Joshua B. Tenenbaum, and Egon Pasztor. An examplebased approach to style translation for line drawings. Technical Report TR99-11, MERL, February 1999.
	19	D. Gentner. Structure mapping: A theoretical framework for analogy. Cognitive Science, 7(2):155-170, 1983.
	20	Allen Gersho , Robert M. Gray, Vector quantization and signal compression, Kluwer Academic Publishers, Norwell, MA, 1991
	21	Paul Haeberli, Paint by numbers: abstract image representations, Proceedings of the 17th annual conference on Computer graphics and interactive techniques, p.207-214, September 1990, Dallas, TX, USA
	22	J. Hamel and T. Strothotte. Capturing and re-using rendition styles for nonphotorealistic rendering. Computer Graphics Forum, 18(3):173-182, September 1999.
	23	David J. Heeger , James R. Bergen, Pyramid-based texture analysis/synthesis, Proceedings of the 22nd annual conference on Computer graphics and interactive techniques, p.229-238, September 1995 [doi>10.1145/218380.218446]
	24	Aaron Hertzmann, Painterly rendering with curved brush strokes of multiple sizes, Proceedings of the 25th annual conference on Computer graphics and interactive techniques, p.453-460, July 1998 [doi>10.1145/280814.280951]
	25	Aaron Hertzmann. Algorithms for Rendering in Artistic Styles. PhD thesis, New York University, May 2001.
	26	Aaron Hertzmann , Denis Zorin, Illustrating smooth surfaces, Proceedings of the 27th annual conference on Computer graphics and interactive techniques, p.517-526, July 2000 [doi>10.1145/344779.345074]
	27	Youichi Horry , Ken-Ichi Anjyo , Kiyoshi Arai, Tour into the picture: using a spidery mesh interface to make animation from a single image, Proceedings of the 24th annual conference on Computer graphics and interactive techniques, p.225-232, August 1997 [doi>10.1145/258734.258854]
	28	William James. The Principles of Psychology. 1890.
	29	Bela Julesz. Textons, the elements of texture perception, and their interactions. Nature, 290:91-97, 1981.
	30	Allison W. Klein , Wilmot Li , Michael M. Kazhdan , Wagner T. Corrêa , Adam Finkelstein , Thomas A. Funkhouser, Non-photorealistic virtual environments, Proceedings of the 27th annual conference on Computer graphics and interactive techniques, p.527-534, July 2000 [doi>10.1145/344779.345075]
	31	Arthur Koestler. The Act of Creation. Picador, London, 1964.
	32	Michael A. Kowalski , Lee Markosian , J. D. Northrup , Lubomir Bourdev , Ronen Barzel , Loring S. Holden , John F. Hughes, Art-based rendering of fur, grass, and trees, Proceedings of the 26th annual conference on Computer graphics and interactive techniques, p.433-438, July 1999 [doi>10.1145/311535.311607]
	33	G. Lakoff and M. Johnson. Metaphors we live by. University of Chicago Press, Chicago, IL, 1980.
	34	Thomas Leung , Jitendra Malik, Recognizing Surfaces Using Three-Dimensional Textons, Proceedings of the International Conference on Computer Vision-Volume 2, p.1010, September 20-25, 1999
	35	Jitendra Malik , Serge Belongie , Jianbo Shi , Thomas Leung, Textons, Contours and Regions: Cue Integration in Image Segmentation, Proceedings of the International Conference on Computer Vision-Volume 2, p.918, September 20-25, 1999
	36	Barbara J. Meier, Painterly rendering for animation, Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, p.477-484, August 1996 [doi>10.1145/237170.237288]
	37	P. Perona , J. Malik, Scale-Space and Edge Detection Using Anisotropic Diffusion, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.12 n.7, p.629-639, July 1990 [doi>10.1109/34.56205]
	38	Ferdinand Petrie and John Shaw. The Big Book of Painting Nature in Watercolor. Watson-Guptill Publications, 1990.
	39	Kris Popat and Rosalind W. Picard. Cluster-based probability model and its application to image and texture processing. IEEE Trans. on Image Processing, 6(2):268-284, February 1997.
	40	Javier Portilla , Eero P. Simoncelli, A Parametric Texture Model Based on Joint Statistics of Complex Wavelet Coefficients, International Journal of Computer Vision, v.40 n.1, p.49-70, Oct. 2000 [doi>10.1023/A:1026553619983]
	41	Michael P. Salisbury , Sean E. Anderson , Ronen Barzel , David H. Salesin, Interactive pen-and-ink illustration, Proceedings of the 21st annual conference on Computer graphics and interactive techniques, p.101-108, July 1994 [doi>10.1145/192161.192185]
	42	Michael P. Salisbury , Michael T. Wong , John F. Hughes , David H. Salesin, Orientable textures for image-based pen-and-ink illustration, Proceedings of the 24th annual conference on Computer graphics and interactive techniques, p.401-406, August 1997 [doi>10.1145/258734.258890]
	43	Arno Schödl , Richard Szeliski , David H. Salesin , Irfan Essa, Video textures, Proceedings of the 27th annual conference on Computer graphics and interactive techniques, p.489-498, July 2000 [doi>10.1145/344779.345012]
	44	K. Schunn and K. Dunbar. Priming, Analogy and Awareness in complex reasoning. Memory and Cognition, 24:271-284, 1996.
	45	E. P. Simoncelli , W. T. Freeman, The steerable pyramid: a flexible architecture for multi-scale derivative computation, Proceedings of the 1995 International Conference on Image Processing (Vol. 3)-Volume 3, p.3444, October 23-26, 1995
	46	Oleg Veryovka and John W. Buchanan. Comprehensive Halftoning of 3D Scenes. Computer Graphics Forum, 18(3):13-22, September 1999.
	47	Oleg Verevka , John W. Buchanan, Halftoning with image-based dither screens, Proceedings of the 1999 conference on Graphics interface '99, p.167-174, September 1999, Kingston, Ontario, Canada
	48	B. Wandell. Foundations of Vision. Sinauer Associates Inc., 1995.
	49	Li-Yi Wei , Marc Levoy, Fast texture synthesis using tree-structured vector quantization, Proceedings of the 27th annual conference on Computer graphics and interactive techniques, p.479-488, July 2000 [doi>10.1145/344779.345009]
	50	Georges Winkenbach , David H. Salesin, Computer-generated pen-and-ink illustration, Proceedings of the 21st annual conference on Computer graphics and interactive techniques, p.91-100, July 1994 [doi>10.1145/192161.192184]
	51	Georges Winkenbach , David H. Salesin, Rendering parametric surfaces in pen and ink, Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, p.469-476, August 1996 [doi>10.1145/237170.237287]
	52	Patrick H. Winston, Learning and reasoning by analogy, Communications of the ACM, v.23 n.12, p.689-703, Dec. 1980 [doi>10.1145/359038.359042]
	53	Daniel N. Wood , Adam Finkelstein , John F. Hughes , Craig E. Thayer , David H. Salesin, Multiperspective panoramas for cel animation, Proceedings of the 24th annual conference on Computer graphics and interactive techniques, p.243-250, August 1997 [doi>10.1145/258734.258859]
	54	Song Chun Zhu , Yingnian Wu , David Mumford, Filters, Random Fields and Maximum Entropy (FRAME): Towards a Unified Theory for Texture Modeling, International Journal of Computer Vision, v.27 n.2, p.107-126, Mar 1998 [doi>10.1023/A:1007925832420]

Collaborative Colleagues:

Aaron Hertzmann: colleagues

Charles E. Jacobs: colleagues

Nuria Oliver: colleagues

Brian Curless: colleagues

David H. Salesin: colleagues