The statistical significance of the MUC-4 results

ABSTRACT

The MUC-4 scores of recall, precision, and the F-measures are used to measure the performance of the participating systems. The differences in the scores between any two systems may be due to chance or may be due to a significant difference between the two systems. To rule out the possibility that the difference is due to chance, statistical hypothesis testing is used. The method of hypothesis testing used is a computationally-intensive method known as approximate randomization. The method and the statistical significance of the results for the two MUC-4 test sets, TST3 and TST4, will be discussed in this paper.

REFERENCES

• 1: Noreen, E. W. (1989) Computer Intensive Methods for Testing Hypotheses: An Introduction. New York: John Wiley & Sons.
• 2: Efron, B. and R. Tibshirani (1991) "Statistical Data Analysis in the Computer Age" Science, vol. 253, pp. 390--395.
• 3: Nancy Chinchor , David D. Lewis , Lynette Hirschman, Evaluating message understanding systems: an analysis of the third message understanding conference (MUC-3), Computational Linguistics, v.19 n.3, p.409-449, September 1993