Comparing human and machine performance for natural language information extraction: results for English microelectronics from the MUC-5 evaluation

ABSTRACT

In evaluating the state of technology for extracting information from natural language text by machine, it is valuable to compare the performance of machine extraction systems with that achieved by humans performing the same task. The purpose of this paper is to present some results from a comparative study of human and machine performance for one of the information extraction tasks used in the MUC-5/Tipster evaluation that can help assess the maturity and applicability of the technology.

REFERENCES

• 1: Will, Craig A. Comparing Human and Machine Performance for Natural Language Information Extraction: Results from the Tipster Text Evaluation. Presented at the Tipster Phase I conference, Fredericksburg, VA, September 23-26, 1993. Proceedings of the TIPSTER Text Program, Phase One. San Mateo, CA: Morgan Kaufmann, Inc., 1994.
• 2: Beth M. Sundheim, Overview of the fourth message understanding evaluation and conference, Proceedings of the 4th conference on Message understanding, June 16-18, 1992, McLean, Virginia [doi>10.3115/1072064.1072066]
• 3: Will, Craig A., and Reeker, Larry H. Issues in the Design of Human-Machine Systems for Natural Language Information Extraction. Presented at the 18-month Tipster meeting, February 22-24, 1993, Williamsburg, VA. Paper available from authors.
• 4: Onyshkevych, Boyan A. Template Design for Information Extraction. This proceedings.
• 5: Carlson, Lynn, Onyshkevych, Boyan, and Okurowski, Mary Ellen. Corpora and Data Preparation. This proceedings.
• 6: Onyshkevych, Boyan, Okurowski, Mary Ellen, and Carlson, Lynn. Tasks, Domains, and Languages. This proceedings.
• 7: Nancy Chinchor , Beth Sundheim, MUC-5 evaluation metrics, Proceedings of the 5th conference on Message understanding, August 25-27, 1993, Baltimore, Maryland [doi>10.3115/1072017.1072026]
• 8: SAIC. Tipster/MUC-5 Scoring System User's Manual. Version 4.3, August 16, 1993.