A system for understanding imaged infographics and its applications

ABSTRACT

Information graphics, or infographics, are visual representations of information, data or knowledge. Understanding of infographics in documents is a relatively new research problem, which becomes more challenging when infographics appear as raster images. This paper describes technical details and practical applications of the system we built for recognizing and understanding imaged infographics located in document pages. To recognize infographics in raster form, both graphical symbol extraction and text recognition need to be performed. The two kinds of information are then auto-associated to capture and store the semantic information carried by the infographics. Two practical applications of the system are introduced in this paper, including supplement to traditional optical character recognition (OCR) system and providing enriched information for question answering (QA). To test the performance of our system, we conducted experiments using a collection of downloaded and scanned infographic images. Another set of scanned document pages from the University of Washington document image database were used to demonstrate how the system output can be used by other applications. The results obtained confirm the practical value of the system.

REFERENCES

• 1: S. Bergler, C. Y. Suen, C. Nadal, N. Nobile, B. Waked, and A. Bloch, Logical block labeling of diverse types of document images, DLIA'99, 1999, 4470--4475.
• 2: T. M. Breuel, Layout Analysis based on Text Line Segment Hypotheses, DLIA'03, Edinburgh, Scotland, August, 2003.
• 3: S. Carberry, S. Elzer, N. Green, K. McCoy and D. Chester, Extending Document Summarization to Information Graphics, Proc. of the ACL Workshop on Text Summarization, 2004.
• 4: S. Carberry, S. Elzer, N. Green, K. McCoy, and D. Chester, Understanding Information Graphics: A Discourse-Level Problem, Proc. of SigDial, 2003, 1--12.
• 5: William S. Cleveland, The elements of graphing data, Wadsworth Publ. Co., Belmont, CA, 1985
• 6: William S. Cleveland, Visualizing Data, Hobart Press, 1993
• 7: Hang Cui , Min-Yen Kan , Tat-Seng Chua, Unsupervised learning of soft patterns for generating definitions from online news, Proceedings of the 13th international conference on World Wide Web, May 17-20, 2004, New York, NY, USA [doi>10.1145/988672.988686]
• 8: Andrew Fitzgibbon , Maurizio Pilu , Robert B. Fisher, Direct Least Square Fitting of Ellipses, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.21 n.5, p.476-480, May 1999 [doi>10.1109/34.765658]
• 9: Robert P. Futrelle , Ioannis A. Kakadiaris , Jeff Alexander , Catherine M. Carriero , Nikos Nikolakis , Joseph M. Futrelle, Understanding Diagrams in Technical Documents, Computer, v.25 n.7, p.75-78, July 1992 [doi>10.1109/2.144445]
• 10: L. Gillard, P. Bellot, M. El-Bèze, Evaluations of Question Answering and Evaluations of the Evaluation, The fifth Int. Conf. on Language Resources and Evaluation, LREC 2006, Genoa, Italy, 24-26 May 2006.
• 11: W. Huang, C. L. Tan and W. K. Leow, Model based chart image recognition, 6th Int. Workshop on Graphics Recognition, GREC'03, 2003, 87--99.
• 12: Weihua Huang , Siqi Zong , Chew Lim Tan, Chart Image Classification Using Multiple-Instance Learning, Proceedings of the Eighth IEEE Workshop on Applications of Computer Vision, p.27, February 21-22, 2007 [doi>10.1109/WACV.2007.17]
• 13: Rohit J. Kate , Raymond J. Mooney, Using string-kernels for learning semantic parsers, Proceedings of the 21st International Conference on Computational Linguistics and the 44th annual meeting of the Association for Computational Linguistics, p.913-920, July 17-18, 2006, Sydney, Australia [doi>10.3115/1220175.1220290]
• 14: J. H. Larkin and H. A. Simon. Why a Diagram is (sometimes) Worth Ten Thousand Words. In Cognitive Science, Vol. 11, No. 1, 1987, 65--100.
• 15: J. Ross Quinlan, C4.5: programs for machine learning, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1993
• 16: Deepak Ravichandran , Eduard Hovy, Learning surface text patterns for a Question Answering system, Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, July 07-12, 2002, Philadelphia, Pennsylvania [doi>10.3115/1073083.1073092]
• 17: Karl Tombre , Salvatore Tabbone , Loïc Pélissier , Bart Lamiroy , Philippe Dosch, Text/Graphics Separation Revisited, Proceedings of the 5th International Workshop on Document Analysis Systems V, p.200-211, August 19-21, 2002
• 18: F. Wang and M. Y. Kan, NPIC: Hierarchical synthetic image classification using image search and generic features, Proc. of Conf. on Image and Video Retrieval, 2006, 473--482.
• 19: Jinxi Xu , Ralph Weischedel , Ana Licuanan, Evaluation of an extraction-based approach to answering definitional questions, Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval, July 25-29, 2004, Sheffield, United Kingdom [doi>10.1145/1008992.1009064]
• 20: Naoko Yokokura , Toyohide Watanabe, Layout-Based Approach for Extracting Constructive Elements of Bar-Charts, Selected Papers from the Second International Workshop on Graphics Recognition, Algorithms and Systems, p.163-174, August 22-23, 1997
• 21: J. Yu, J. Hunter, E. Reiter and S. Sripada, Recognizing visual patterns to communicate gas turbine time-series data, ES2002, 2002, 105--118.
• 22: Bo Yuan , Chew Lim Tan, Multi-Level Component Grouping Algorithm and Its Applications, Proceedings of the Eighth International Conference on Document Analysis and Recognition, p.1178-1181, August 31-September 01, 2005 [doi>10.1109/ICDAR.2005.164]
• 23: Y. Zheng, C. Liu, X. Ding and S. Pan, A Form Frame-Line Detection Algorithm Based on Directional Single-Connected Chain, Journal of Software, Vol. 13, 2002, 790--796.
• 24: Y. Zhou and C. L. Tan, Hough-based Model for Recognizing Bar Charts in Document Images, SPIE conference on Document image and retrieval, 2001.
• 25: Y. Zhou and C. L. Tan, Learning-based scientific chart recognition, 4th IAPR Int. Workshop on Graphics Recognition, GREC'01, 2001, 482--492.
• 26: Y. Zhou and C. L. Tan, Coordinate systems reconstruction for graphical documents by Hough feature clustering and geometric analysis, Int. Conf. on Pattern Recognition, ICPR'04, Cambridge, UK, 23-26 Aug 2004.

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.7 DOCUMENT AND TEXT PROCESSING
      I.7.5 Document Capture
          Subjects: Graphics recognition and interpretation

Additional Classification:
  I. Computing Methodologies
  I.7 DOCUMENT AND TEXT PROCESSING
      I.7.5 Document Capture
          Subjects: Document analysis

General Terms:
Design, Experimentation

Keywords:
applications, association of text and graphics, document image understanding, infographics