Semantics and CBIR: a medical imaging perspective

ABSTRACT

Medical CBIR (content-based image retrieval) applications pose unique challenges but at the same time offer many new opportunities. On one hand, while one can easily understand news or sports videos, a medical image is often completely incomprehensible to untrained eyes. On the other hand, semantics in the medical domain is much better defined and there is a vast accumulation of formal knowledge representations that could be exploited to support semantic search for any specialty areas in medicine.

In this paper, however, we will not dwell on any one particular specialty area, but rather address the question of how to support scalable semantic search across the whole of medical CBIR field: what are the advantages to take and gaps to fill, what are the key enabling technologies, and the critical success factor from an industrial point of view.

In terms of enabling technologies, we discuss three aspects: 1. anatomical, disease, and contextual semantics, and their representations using ontologies; 2. scalable image analysis and tagging algorithms; and 3. ontological reasoning and its role in guiding and improving image analysis and retrieval.

More specifically, for ontological representation of medical imaging semantics, we discuss the potential use of FMA, RadLex, ICD, and AIM. For scalable image analysis we present a learning-based anatomy detection and segmentation framework using distribution-free priors. It is easily adaptable to different anatomies and different imaging modalities.

REFERENCES

• 1: S. Antani, L. R. Long, and G. R. Thoma. A biomedical info. system for combined content-based retrieval of spine x-ray images and associated text information. In Proc. Indian Conf. Comp. Vision, Graphics, & Image Proc., 2002.
• 2: M. Aurnhammer, P. Hanappe, and L. Steels. Integrating collaborative tagging and emergent semantics for image retrieval. In Proc. Collab. Web Tagging Workshop, 2006.
• 3: Bing Bai , Paul Kantor , Ali Shoukoufandeh , Deborah Silver, fMRI Brain Image Retrieval Based on ICA Components, Proceedings of the Eighth Mexican International Conference on Current Trends in Computer Science, p.10-17, September 24-28, 2007 [doi>10.1109/ENC.2007.18]
• 4: T. Berners-Lee, J. Hendler, and O. Lassila. The Semantic Web -- A new form of Web content that is meaningful to computers will unleash a revolution of new possibilities. Scientific American, May 2001.
• 5: O. Bodenreider. The Unified Medical Language System (UMLS): Integrating biomedical terminology. Nucleic Acids Research, 32, 2004.
• 6: J. G. Bosch, S. C. Mitchell, P. F. Lelieveldt, F. Nijland, O. Kamp, M. Sonka, and J. H. C. Reiber. Automatic segmentation of echocardiographic sequences by active appearance motion models. IEEE Trans. Medical Imaging, 21(11):1374--1383, 2002.
• 7: D. Brickley and R. Guha. RDF vocabulary description language 1.0: RDF Schema, February 2004.
• 8: D. G. Brown, The Evaluation of Computer-Aided Diagnosis Systems: An FDA Perspective, Proceedings of the 30th on Applied Imagery Pattern Recognition Workshop, p.17, October 10-12, 2001
• 9: P. Buitelaar, M. Sintek, and M. Kiesel. A lexicon model for multilingual/multimedia ontologies. Proc. 3rd European Semantic Web Conference (ESWC06), June 2006.
• 10: W. Cai, D. Feng, and R. Fulton. Content-based retrieval of dynamic PET functional images. IEEE Trans. Info. Tech. in Biomedicine, 4(2):152--158, 2000.
• 11: J. Carazo and H. Stelter. The bioimage database project:organizing multidemensional biological images in an object-relational database. J. of Struct. Biol., 125:97--102, 1999.
• 12: S. Castano, A. Ferrara, and G. Hess. Discovery-driven ontology evolution. Proc. 3rd Italian Semantic Web Workshop (SWAP06), December 2006.
• 13: R. Chbeir, Y. Amghar, and A. Flory. MIMS: A prototype for medical image retrieval. In Recherche d'Informations Assistee par Ordinateur, 2000.
• 14: C. I. Christodoulou, E. Kyriacou, C. S. Pattichis, and A. Nicolaides. Multiple feature extraction for content-based image retrieval of carotid plaque ultrasound images. In Proc. Int'l Special Topic Conf. Info. Tech. in Biomedicine, 2006.
• 15: Wesley W. Chu , Alfonso F. Cárdenas , Ricky K. Taira, KMeD: a knowledge-based multimedia medical distributed database system, Information Systems, v.20 n.2, p.75-96, April 1995 [doi>10.1016/0306-4379(95)98556-S]
• 16: Wesley W. Chu , Chih-Cheng Hsu , Alfonso F. Cárdenas , Ricky K. Taira, Knowledge-Based Image Retrieval with Spatial and Temporal Constructs, IEEE Transactions on Knowledge and Data Engineering, v.10 n.6, p.872-888, November 1998 [doi>10.1109/69.738355]
• 17: T. Cootes and C. Taylor. Active shape models-'smart snakes'. In Proc. British Machine Vision Conf., pages 266--275, 1992.
• 18: T. Deselaers, D. Keysers, and H. Ney. FIRE -- flexible image retrieval engine: ImageCLEF 2004 evaluation. In CLEF 2004, LNCS 3491, pages 688--698, September 2004.
• 19: James C. Duncan , Nicholas Ayache, Medical Image Analysis: Progress over Two Decades and the Challenges Ahead, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.22 n.1, p.85-106, January 2000 [doi>10.1109/34.824822]
• 20: M. Giger, Huo, Vyborny, Lan, Bonta, Horsch, Nishikawa, and Rosenbourgh. Intelligent CAD workstation for breast imaging using similarity to known lesions and multiple visual prompt aids. In Proc. SPIE Medical Imaging, 2002.
• 21: Tristan Glatard , Johan Montagnat , Isabelle E. Magnin, Texture based medical image indexing and retrieval: application to cardiac imaging, Proceedings of the 6th ACM SIGMM international workshop on Multimedia information retrieval, October 15-16, 2004, New York, NY, USA [doi>10.1145/1026711.1026734]
• 22: Thomas R. Gruber, Toward principles for the design of ontologies used for knowledge sharing, International Journal of Human-Computer Studies, v.43 n.5-6, p.907-928, Nov./Dec. 1995 [doi>10.1006/ijhc.1995.1081]
• 23: W. Hong, B. Georgescu, X. S. Zhou, S. Krishnan, Y. Ma, and D. Comaniciu. Database-guided simultaneous multi-slice 3D segmentation for volumetric data. In European Conf. Computer Vision, volume 3954, pages 397--409, May 2006.
• 24: T. Huang, C. Dagli, S. Rajaram, E. Chang, M. Mandel, G. Poliner, and D. Ellis. Active learning for interactive multimedia retrieval. Proc. IEEE, 96, 2008.
• 25: Y. Huang, S. Kuo, C. Chang, Y. Liu, W. Moon, and D. Chen. Image retrieval with principal component analysis for breast cancer diagnosis on various ultrasonic systems. Ultrasound in Obstetrics and Gynecology, 26(5):558--566, 2005.
• 26: A. Jerebko, G. Schmidt, X. Zhou, J. Bi, V. Anand, J. Liu, S. Schoenberg, I. Schmuecking, B. Kiefer, and A. Krishnan. Computer-aided detection of skeletal metastases in MRI STIR imaging of the spine. In Proc. Info. Processing in Medical Imaging (IPMI), 2007.
• 27: C. J. Kahn and C. Thao. GoldMiner: A radiology image search engine. Am. J. Roentgenology, 188, 2007.
• 28: Philip (Flip) Korn , Nicholas Sidiropoulos , Christos Faloutsos , Eliot Siegel , Zenon Protopapas, Fast and Effective Retrieval of Medical Tumor Shapes, IEEE Transactions on Knowledge and Data Engineering, v.10 n.6, p.889-904, November 1998 [doi>10.1109/69.738356]
• 29: J. Kuo, R. Chang, C. Lee, W. Moon, and D. Chen. Retrieval technique for the diagnosis of solid breast tumors on sonogram. Ultrasound Med Biol., 28:903--909, 2002.
• 30: D. Kwak, B. Kim, O. Yoon, C. Park, J. Won, and K. Park. Content-based ultrasound image retrieval using a coarse to fine approach. Annals NY Acad. Sci., 980:212--224, 2002.
• 31: C. Langlotz. Radlex: A new method for indexing online educational materials. RadioGraphics, (26):1595--1597, 2006.
• 32: C. Le Bozec, M. C. Jaulent, E. Zapletal, D. Heudes, and P. Degoulet. IDEM: A Web application of case-based reasoning in histopathology. Comput Biol Med., 28(5):473--87, 1998.
• 33: T. Lehmann, M. Güld, C. Thies, B. Fischer, K. Spitzer, D. Keysers, H. Ney, M. Kohnen, H. Schubert, and B. Wein. Content-based image retrieval in medical applications. Methods Inf. Med., 43, 2004.
• 34: Douglas B. Lenat, CYC: a large-scale investment in knowledge infrastructure, Communications of the ACM, v.38 n.11, p.33-38, Nov. 1995 [doi>10.1145/219717.219745]
• 35: C. T. Liu, P. L. Tai, A.-J. Chen, C.-H. Peng, and J. S. Wang. A content-based scheme for CT lung image retrieval. In Proc. Int'l Conf. Multimedia & Expo, 2000.
• 36: Y. Liu , F. Dellaert, A Classification Based Similarity Metric for 3D Image Retrieval, Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, p.800, June 23-25, 1998
• 37: L. R. Long, L. E. Berman, and G. R. Thoma. Prototype client/server application for biomedical text/image retrieval on the internet. In Storage and Retrieval for Image and Video Databases (SPIE), pages 362--372, 1996.
• 38: M. Mattie, L. Staib, E. Stratmann, H. D. Tagare, J. Duncan, and P. L. Miller. PathMaster: Content-based cell image retrieval using automated feature extraction. J. Amer. Medical Informatics Assoc., 7:404--415, 2000.
• 39: D. L. McGuinness and F. van Harmelen. OWL web ontology language overview, February 2004.
• 40: A. Mechouche, C. Golbreich, and B. Gibaud. Towards an hybrid system using an ontology enriched by rules for the semantic annotation of brain MRI images. In Lecture Notes Computer Sci., volume 4524, pages 219--228, June 2007.
• 41: H. Müller, A. Rosset, A. Garcia, J.-P. Vallée, and A. Geissbuhler. Informatics in radiology (inforad): Benefits of content-based visual data access in radiology. RadioGraphics, 19:33--54, 2005.
• 42: A. Mojsilovic and J. Gomes. Semantic based categorization, browsing and retrieval in medical image databases. In Proc. Int'l Conf. Image Proc., 2002.
• 43: H. Müller, N. Michoux, D. Bandon, and A. Geissbuhler. A review of content-based image retrieval systems in medical applications - clinical benefits and future directions. Int'l J. of Medical Informatics, 73(1):1--23, 2004.
• 44: S. Orphanoudakis, C. Chronaki, and S. Kostomanolakis. I2Cnet: A system for the indexing. storage and retrieval of medical images by content. Med. Informatics, 19:109--122, 1994.
• 45: G. T. Papadopoulosa, V. Mezaris, S. Dasiopoulou, and I. Kompatsiaris. Semantic image analysis using a learning approach and spatial context. In Proc. 1st Int'l Conf. Semantics & digital Media Tech., December 2006.
• 46: S. Patwardhan, A. Dhawan, and P. Relue. Classification of melanoma using tree structured wavelet transforms. Computer Methods and Programs in Biomedicine, 72(3):223--239, 2003.
• 47: R. Pompl, W. Bunk, A. Horsch, W. Stolz, W. Abmayr, W. Brauer, A. GläSSl, and G. Morfill. MELDOQ: Ein system zur unterstützung der früherkennung des malignen melanoms durch digitale bildverarbeitung. In Proc. Workshop Bildverarbeitung für die Medizin, 2000.
• 48: G. Robinson, H. Tagare, J. Duncan, and C. Jaffe. Medical image collection indexing: shape-based retrieval using KD-trees. Computerized Medical Imaging and Graphics, 20:209--217, 1996.
• 49: M. Romanelli, P. Buitelaar, and M. Sintek. Modeling linguistic facets of multimedia content for semantic annotation. In Proc. Int'l Conf. Semantics & digital Media Tech., December 2007.
• 50: Cornelius Rosse , José L. V. Mejino, Jr., A reference ontology for biomedical informatics: the foundational model of anatomy, Journal of Biomedical Informatics, v.36 n.6, p.478-500, December 2003 [doi>10.1016/j.jbi.2003.11.007]
• 51: D. Rubin, P. Mongkolwat, V. Kleper, K. Supekar, and D. Channin. Medical imaging on the Semantic Web: Annotation and image markup. In AAAI Spring Symposium Series, Semantic Scientific Knowledge Integration, 2008.
• 52: Simone Santini , Amarnath Gupta , Ramesh Jain, Emergent Semantics through Interaction in Image Databases, IEEE Transactions on Knowledge and Data Engineering, v.13 n.3, p.337-351, May 2001 [doi>10.1109/69.929893]
• 53: P. Schmidt-Saugeon, J. Guillod, and J.-P. Thiran. Towards a computer-aided diagnosis system for pigmented skin lesions. Computerized Med. Imaging & Graphics, 27:65--78, 2003.
• 54: F. Schnorrenberg , C. S. Pattichis , C. N. Schizas , K. Kyriacou, Content-based retrieval of breast cancer biopsy slides, Technology and Health Care, v.8 n.5, p.291-297, Dec. 1,2000
• 55: M. Sermesant, C. Forest, X. Pennec, H. Delingette, and N. Ayache. Deformable biomechanical models: Application to 4D cardiac image analysis. Med. Image Anal, 7, 2003.
• 56: Chi-Ren Shyu , Carla E. Brodley , Avinash C. Kak , Akio Kosaka , Alex M. Aisen , Lynn S. Broderick, ASSERT: a physician-in-the-loop content-based retrieval system for HRCT image databases, Computer Vision and Image Understanding, v.75 n.1-2, p.111-132, July/Aug. 1999 [doi>10.1006/cviu.1999.0768]
• 57: E. Sirin and B. Parsia. Pellet: An OWL DL reasoner. In V. Haarslev and R. Möller, editors, Description Logics, volume 104 of CEUR Workshop Proc., 2004.
• 58: Arnold W. M. Smeulders , Marcel Worring , Simone Santini , Amarnath Gupta , Ramesh Jain, Content-Based Image Retrieval at the End of the Early Years, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.22 n.12, p.1349-1380, December 2000 [doi>10.1109/34.895972]
• 59: M. Stearns, C. Price, K. Spackman, and A. Wang. SNOMED clinical terms: overview of the development process and project status. In Proc. Annual Symp. of Amer. Medical Informatics Asso., 2001.
• 60: L. Su, B. Sharp, and C. Chibelushi. Knowledge-based image understanding: A rule-based production system for X-ray segmentation. In Proc. Int'l Conf. Enterprise Info. System, volume 1, pages 530--533, Spain, April 2002.
• 61: H. Tang, R. Hanka, and H. Ip. Histological image retrieval based on semantic content analysis. IEEE Trans. Info. Technology in Biomedicine, 7(1):26--36, 2003.
• 62: Simon Tong , Edward Chang, Support vector machine active learning for image retrieval, Proceedings of the ninth ACM international conference on Multimedia, September 30-October 05, 2001, Ottawa, Canada [doi>10.1145/500141.500159]
• 63: Z. Tu, X. S. Zhou, L. Bogoni, A. Barbu, and D. Comaniciu. Probabilistic 3D polyp detection in CT images: The role of sample alignment. IEEE CVPR, 2:1544--1551, May 2006.
• 64: M. Uschold and M. Grüninger. Ontologies: Principles, methods and applications. Knowledge Engineering Review, 11(2), 1996.
• 65: J. Vompras. Towards adaptive ontology-based image retrieval. In 17th GI-Workshop on the Foundations of Databases, Wörlitz, Germany, pages 148--152, May 2005.
• 66: J. Z. Wang. Pathfinder: Multiresolution region-based searching of pathology images using IRM. In Proc AMIA Symp., pages 883--887, 2000.
• 67: P. Wennerberg, P. Buitelar, and S. Zillner. Towards a human anatomy data set for query pattern mining based on Wikipedia and domain semantic resources. In Proc. Int'l Conf. Langu. Res. & Eval., Morocco, May 2008.
• 68: Y. Zhan, X. S. Zhou, and A. Krishnan. An information theoretic view of the scheduling problem in whole-body CAD. In Proc. SPIE Medical Imaging, 2008.
• 69: Xiang Sean Zhou , Dorin Comaniciu , Alok Gupta, An Information Fusion Framework for Robust Shape Tracking, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.27 n.1, p.115-129, January 2005 [doi>10.1109/TPAMI.2005.3]
• 70: X. S. Zhou and T. S. Huang. Relevance feedback in image retrieval: A comprehensive review. Multimedia Sys., 8, 2003.
• 71: X. S. Zhou, Y. Rui, and T. S. Huang. Exploration of Visual Data. Kluwer Academic Publishers, 2003.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL

General Terms:
Algorithms

Keywords:
image retrieval, medical image analysis, ontology, semantics