Extracting significant time varying features from text

ABSTRACT

We propose a simple statistical model for the frequency of occurrence of features in a stream of text. Adoption of this model allows us to use classical significance tests to filter the stream for interesting events. We tested the model by building a system and running it on a news corpus. By a subjective evaluation, the system worked remarkably well: almost all of the groups of identified tokens corresponded to news stories and were appropriately placed in time. A preliminary objective evaluation was also used to measure the quality of the system and it showed some of the weaknesses and the power of our approach.

REFERENCES

• 1: Facts on File, 1996. Facts on File, New York, 1997.
• 2: J. Allan, J. Carbonell, G. Doddington, J. Yamron, and Y. Yang. Topic detection and tracking pilot study: Final report. In Proceedings of the DARPA Broadcast News 2kanscription and Understanding Workshop, pages 194-218, 1998.
• 3: R. B. Allen. Timelines as information system interfaces. In Proceedings International Symposium on Digital Libraries, pages 175-180, Tsukuba, Japan, 1995.
• 4: Yvonne M. M. Bishop, Stephen E. Feinberg, and Paul W. Holland. Discrete multivariate analysis: theory and practice. MIT Press, Cambridge, Massachusetts, 1974.
• 5: Ido Dagan and Ronen Feldman. Keyword-based browsing and analysis of large document sets. In Proceedings o/the Symposium on Document Analysis and Information Retrieval (SDAiR-96), Las Vegas, Nevada, 1996.
• 6: David Fisher , Stephen Soderland , Fangfang Feng , Wendy Lehnert, Description of the UMass system as used for MUC-6, Proceedings of the 6th conference on Message understanding, November 06-08, 1995, Columbia, Maryland [doi>10.3115/1072399.1072412]
• 7: Robin L. Kultberg. Dynamic timelines: Visualizing historical information in three dimensions. Master's thesis, Massachusetts Institute of Technology Media Laboratory, 1995.
• 8: Eugene L. Margulis, Modelling documents with multiple Poisson distributions, Information Processing and Management: an International Journal, v.29 n.2, p.215-227, March-April 1993 [doi>10.1016/0306-4573(93)90005-X]
• 9: Ron Papka, James Allan, and Victor Lavrenko. Umass approaches to detection and tracking at TDT2. In Proceedings of the DARPA Broadcast Workshop, 1999.
• 10: Catherine Plaisant , Brett Milash , Anne Rose , Seth Widoff , Ben Shneiderman, LifeLines: visualizing personal histories, Proceedings of the SIGCHI conference on Human factors in computing systems: common ground, p.221-ff., April 13-18, 1996, Vancouver, British Columbia, Canada [doi>10.1145/238386.238493]
• 11: Mark Sanderson , Bruce Croft, Deriving concept hierarchies from text, Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval, p.206-213, August 15-19, 1999, Berkeley, California, United States [doi>10.1145/312624.312679]
• 12: J. Xu , J. Broglio , B. Croft, The Design and Implementation of a Part of Speech Tagger for English, University of Massachusetts, Amherst, MA, 1994

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.7 Natural Language Processing
          Subjects: Text analysis

Additional Classification:
  G. Mathematics of Computing

  H. Information Systems
  H.2 DATABASE MANAGEMENT
      H.2.4 Systems
          Subjects: Textual databases

  I. Computing Methodologies
  I.4 IMAGE PROCESSING AND COMPUTER VISION
      I.4.8 Scene Analysis
          Subjects: Time-varying imagery

General Terms:
Design, Documentation, Management, Measurement, Performance, Theory