Tipster/MUC-5: information extraction system evaluation

ABSTRACT

Three information extraction system evaluations using Tipster data were conducted in the context of Phase 1 of the Tipster Text program. Interim evaluations were conducted in September, 1992, and February, 1993; the final evaluation was conducted in July, 1993. The final evaluation included not only the Tipster-supported information extraction contractors but thirteen other participants as well. This evaluation was the topic of the Fifth Message Understanding Conference (MUC-5) in August, 1993. With particular respect to the research and development tasks of the Tipster contractors, the goal of these evaluations has been to assess success in terms of the development of systems to work in both English and Japanese (BBN, GE/CMU, and NMSU/Brandeis) and/or in both the joint ventures and microelectronics domains (BBN, GE/CMU, NMSU/Brandeis, and UMass/Hughes).

REFERENCES

• 1: Carlson, L., et al., Corpora and Data Preparation, in this volume.
• 2: Nancy Chinchor, The statistical significance of the MUC-4 results, Proceedings of the 4th conference on Message understanding, June 16-18, 1992, McLean, Virginia [doi>10.3115/1072064.1072068]
• 3: Chinchor, N., The Statistical Significance of the MUC-5 Results, in this volume.
• 4: Chinchor, N. and Sundheim, B., MUC-5 Evaluation Metrics, in this volume.
• 5: Lynette Hirschman, Comparing MUCK-II and MUC-3: assessing the difficulty of different tasks, Proceedings of the 3rd conference on Message understanding, May 21-23, 1991, San Diego, California [doi>10.3115/1071958.1071962]
• 6: Maiorano, S., An Analysis of the Joint Venture Japanese Text Prototype and Its Effect On System Performance, in Proceedings from the TIPSTER Text Program, Phase One, September 1993, San Mateo: Morgan Kaufmann (to appear).
• 7: Mary Ellen Okurowski, Domain and language evaluation results, Proceedings of the 5th conference on Message understanding, August 25-27, 1993, Baltimore, Maryland [doi>10.3115/1072017.1072024]
• 8: Proceedings of the Fourth Message Understanding Conference (MUC-4), June 1992, San Mateo: Morgan Kaufmann.
• 9: Science Applications International Corporation, Tipster/MUC-5 Scoring System User's Manual, version 4.3, August 1993.
• 10: Beth M. Sundheim , Nancy A. Chinchor, Survey of the Message Understanding Conferences, Proceedings of the workshop on Human Language Technology, March 21-24, 1993, Princeton, New Jersey [doi>10.3115/1075671.1075684]
• 11: Will, C., Comparing Human and Machine Performance for Natural Language Information Extraction: Results from the Tipster Text Evaluation, in Proceedings from the TIPSTER Text Program, Phase One, September 1993, San Mateo: Morgan Kaufmann (to appear).
• 12: Will, C., Comparing Human and Machine Performance for Natural Language Information Extraction: Results for English Microelectronics from the MUC-5 evaluation, in this volume.