Evaluation of DEFINDER: a system to mine definitions from consumer-oriented medical text

ABSTRACT

In this paper we present DEFINDER, a rule-based system that mines cons umer-oriented full text articles in order to extract definitions and the terms they define. This research is part of Digital Library Project at Columbia University, entitled PERSIVAL (PErsonalized Retrieval and Summarization of Image, Video and Language resources) [5]. One goal of the project is to present information to patients in language they can understand. A key component of this stage is to provide accurate and readable lay definitions for technical terms, which may be present in articles of intermediate complexity. The focus of this short paper is on quantitative and qualitative evaluation of the DEFINDER system [3]. Our basis for comparison was definitions from Unified Medical Language System (UMLS), On-line Medical Dictionary (OMD) and Glossary of Popular and Technical Medical Terms (GPTMT). Quantitative evaluations show that DEFINDER obtained 87% precision and 75% recall and reveal the incompleteness of existing resources and the ability of DEFINDER to address gaps. Qualitative evaluation shows that the definitions extracted by our system are ranked higher in terms of user-based criteria of usability and readability than definitions from on-line specialized dictionaries. Thus the output of DEFINDER can be used to enhance existing specialized dictionaries, and also as a key feature in summarizing technical articles for non-specialist users.

REFERENCES

• 1: Marti A. Hearst, Untangling text data mining, Proceedings of the 37th annual meeting of the Association for Computational Linguistics on Computational Linguistics, p.3-10, June 20-26, 1999, College Park, Maryland [doi>10.3115/1034678.1034679]
• 2: Justeson, J. and Katz, S. Technical Terminology: Some Linguistic Properties and an Algorithm for Identification in Text. Natural Language Engineering. Vol 1(1). 1995. pp. 9- 27.
• 3: Klavans J.L., Muresan S. DEFINDER: Rule-Based Methods for the Extraction of Medical Terminology and their Associated Definitions from On-line Text. Proc of AMIA 2000; pp. 1906.
• 4: McCord M.C. The Slot Grammar system. IBM Report; 1991.
• 5: Kathleen R. McKeown , Shih-Fu Chang , James Cimino , Steven Feiner , Carol Friedman , Luis Gravano , Vasileios Hatzivassiloglou , Steven Johnson , Desmond A. Jordan , Judith L. Klavans , André Kushniruk , Vimla Patel , Simone Teufel, PERSIVAL, a system for personalized search and summarization over multimedia healthcare information, Proceedings of the 1st ACM/IEEE-CS joint conference on Digital libraries, p.331-340, January 2001, Roanoke, Virginia, United States [doi>10.1145/379437.379722]
• 6: Siegal, S. and Castellan, N.J. (1988). Non-parametric statistics for the behavioural sciences (2nd Edition). New York: McGraw Hill.
• 7: Zweigenbaum P, Bouaud J, Bachimont B, Charlet J, Seroussi B, Boisvieux JF. From Text to Knowledge: a Unifying Document-Oriented View of Analyzed Medical Language. Proceedings of IMIA WG6. 1997.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL

Additional Classification:
  H. Information Systems
  H.2 DATABASE MANAGEMENT
      H.2.8 Database applications
          Subjects: Data mining
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.1 Content Analysis and Indexing
          Subjects: Dictionaries

  J. Computer Applications
  J.3 LIFE AND MEDICAL SCIENCES
      Subjects: Medical information systems

General Terms:
Design, Documentation, Human Factors, Management, Measurement, Performance, Theory

Keywords:
automatic dictionary creation, medical digital libraries, natural language processing, text data mining