The statistical significance of the MUC-5 results

ABSTRACT

The statistical significance of the results of the MUC-5 evaluation is determined using a computer-intensive method of hypothesis testing known as approximate randomization. The exact method is described in detail in [1] and [2] and has been used as the accepted statistical test for the MUC results since MUC-3. The purpose of the statistical testing is to determine whether the scores of the systems are different by chance or due to a significant difference in the character of the systems.

REFERENCES

• 1: Nancy Chinchor , David D. Lewis , Lynette Hirschman, Evaluating message understanding systems: an analysis of the third message understanding conference (MUC-3), Computational Linguistics, v.19 n.3, p.409-449, September 1993
• 2: Nancy Chinchor, The statistical significance of the MUC-4 results, Proceedings of the 4th conference on Message understanding, June 16-18, 1992, McLean, Virginia [doi>10.3115/1072064.1072068]