Text genre classification with genre-revealing and subject-revealing features

ABSTRACT

Subject or prepositional content has been the focus of most classification research. Genre or style, on the other hand, is a different and important property of text, and automatic text genre classification is becoming important for classification and retrieval purposes as well as for some natural language processing research. In this paper, we present a method for automatic genre classification that is based on statistically selected features obtained from both subject-classified and genre classified training data. The experimental results show that the proposed method outperforms a direct application of a statistical learner often used for subject classification. We also observe that the deviation formula and discrimination formula using document frequency ratios also work as expected. We conjecture that this dual feature set approach can be generalized to improve the performance of subject classification as well.

REFERENCES

• 1: Ivan Bretan, John Dewe, Anders Hallberg, Niklas Wolkert, Jussi Karlgren, "Web-Specific Genre Visualization", Proc. of the 30th Hawaii International Conference on System Science, Jan 1997.
• 2: Johan Dewe, Jussi Karlgren, Ivan Bretan, "Assembling a Balanced Corpus from the Internet", 11th Nordic Conference of Computational Linguistics, pages 100--107, Copenhagen, 1998.
• 3: Andrew Dillon , Barbara A. Gushrowski, Genres and the Web: is the personal home page the first uniquely digital genre?, Journal of the American Society for Information Science, v.51 n.2, p.202-205, Jan. 15, 2000 [doi>10.1002/(SICI)1097-4571(2000)51:2<202::AID-ASI11>3.0.CO;2-R]
• 4: Jussi Karlgren, "Stylistic Variation in an Information Retrieval Experiment", Proc. of the 2nd International Conference on New Methods in Language Processing-NeMLaP, 1996.
• 5: Jussi Karlgren, Ivan Brettan, Johan Dewe, Anders Hallberg, Niklas Wolkert, "Iterative Information Retrieval Using Fast Clustering and Usage-Specific Genres", 8th DELOS Workshop on User Interfaces in Digital Libraries, pages 85--92, 1998.
• 6: Jussi Karlgren , Douglass Cutting, Recognizing text genres with simple metrics using discriminant analysis, Proceedings of the 15th conference on Computational linguistics, August 05-09, 1994, Kyoto, Japan [doi>10.3115/991250.991324]
• 7: Brett Kessler , Geoffrey Numberg , Hinrich Schütze, Automatic detection of text genre, Proceedings of the 35th annual meeting on Association for Computational Linguistics, p.32-38, July 07-12, 1997, Madrid, Spain
• 8: D. Lewis and M. Ringuette, "Compariosn of two learning algorithms for text categorization," Proc. of the 3rd Annual Symposium on Document Analysis and Information Retrieval, 1994.
• 9: Hyo-Jung Oh , Sung Hyon Myaeng , Mann-Ho Lee, A practical hypertext catergorization method using links and incrementally available class information, Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, p.264-271, July 24-28, 2000, Athens, Greece [doi>10.1145/345508.345594]
• 10: E. Stamatatos , N. Fakotakis , G. Kokkinakis, Text genre detection using common word frequencies, Proceedings of the 18th conference on Computational linguistics, July 31-August 04, 2000, Saarbrücken, Germany [doi>10.3115/992730.992763]
• 11: Yiming Yang , Xin Liu, A re-examination of text categorization methods, Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval, p.42-49, August 15-19, 1999, Berkeley, California, United States [doi>10.1145/312624.312647]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.4 Systems and Software

General Terms:
Experimentation, Performance, Verification

Keywords:
classification, idf, statistical method, subject categories, text genre, tf