Take a look at the new version of this page: [ beta version ]. Tell us what you think.

ABSTRACT

This paper presents a new challenge for Web monitoring tools: to build a system that can monitor entire web sites effectively. Such a system could potentially be used to discover "silent news" hidden within corporate web sites. Examples of silent news include reorganizations in the executive team of a company or in the retirement of a product line. ChangeDetector, an implemented prototype, addresses this challenge by incorporating a number of machine learning techniques. The principal backend components of ChangeDetector all rely on machine learning: intelligent crawling, page classification and entity-based change detection. Intelligent crawling enables ChangeDetector to selectively crawl the most relevant pages of very large sites. Classification allows change detection to be filtered by topic. Entity extraction over changed pages permits change detection to be filtered by semantic concepts, such as person names, dates, addresses, and phone numbers. Finally, the front end presents a flexible way for subscribers to interact with the database of detected changes to pinpoint those changes most likely to be of interest.

REFERENCES

• 1: Ackerman, M., Starr B., Pazzani, M. "The Do-I-Care Agent: Effective Social Discovery and Filtering on the Web." In: Proc. of RIAO'97, pp. 17--31.
• 2: Soumen Chakrabarti , Martin van den Berg , Byron Dom, Focused crawling: a new approach to topic-specific Web resource discovery, Proceeding of the eighth international conference on World Wide Web, p.1623-1640, May 1999, Toronto, Canada
• 3: Chen, Y.-F., Douglis, F., Huan, H., Vo, K.-P. "TopBlend: An Efficient Implementation of HtmlDiff in Java." In: Proc. of the WebNet2000 Conference, San
• 4: Chen, Y.-F., Koutsofios, E. "Website news: A website tracking and visualization service." In: Poster Proc. of World Wide Web 8, Toronto, Ontario, Canada, May 1999.
• 5: Dedieu, O. "Pluxy: un proxy Web dynamiquement extensible." In: Proc. of the 1998 NoTeRe Colloquium, Oct. 1998, http://www-sor.inria.fr/publi/PPWDE_notore98.html.
• 6: Fred Douglis , Thomas Ball , Yih-farn Chen , Eleftherios Koutsofios, The AT&T Internet Difference Engine: Tracking and viewing changes on the web, World Wide Web, v.1 n.1, p.27-44, 1998 [doi>10.1023/A:1019243126596]
• 7: Fishkin, K., Bier, E. "WebTracker - a Web Service for tracking documents." In: Proc. of World Wide Web 6, Santa Clara, CA, 1997, http://www.parc.xerox.com/istl/members/ fishkin/doc/webtracker.html.
• 8: Rayid Ghani , Seán Slattery , Yiming Yang, Hypertext Categorization using Hyperlink Patterns and Meta Data, Proceedings of the Eighteenth International Conference on Machine Learning, p.178-185, June 28-July 01, 2001
• 9: Natalie Glance , Jean-Luc Meunier , Pierre Bernard , Damián Arregui, Collaborative document monitoring, Proceedings of the 2001 International ACM SIGGROUP Conference on Supporting Group Work, September 30-October 03, 2001, Boulder, Colorado, USA [doi>10.1145/500286.500313]
• 10: Daniel S. Hirschberg, Algorithms for the Longest Common Subsequence Problem, Journal of the ACM (JACM), v.24 n.4, p.664-675, Oct. 1977 [doi>10.1145/322033.322044]
• 11: Hobbs, Jerry R., Douglas E. Appelt, John Bear, David Israel, Megumi Kameyama, Mark Stickel, and Mabry Tyson. 1996. "FASTUS: A cascaded finite-state transducer for extracting information from natural-language text." In: Finite State Devices for Natural Language Processing. MIT Press, Cambridge, MA.
• 12: Andrew McCallum , Dayne Freitag , Fernando C. N. Pereira, Maximum Entropy Markov Models for Information Extraction and Segmentation, Proceedings of the Seventeenth International Conference on Machine Learning, p.591-598, June 29-July 02, 2000
• 13: Muslea, I. "Extraction Patterns for Information Extraction Tasks: A Survey." In: Proc. of AAAI'99 Workshop on Machine Learning for Information Extraction, Orlando, FL, 1999.
• 14: Nigam, K., Lafferty, J., McCallum, A. "Using Maximum Entropy for Text Classification." In: IJCAI'99 Workshop on Information Filtering, 1999.
• 15: NetMind, http://www.netmind.com/
• 16: Minka, T. "Algorithms for maximum-likelihood logistic regression." Technical Report, 2001. http://www.stat.cmu.edu/~minka/papers/logreg.html
• 17: Jason Rennie , Andrew McCallum, Using Reinforcement Learning to Spider the Web Efficiently, Proceedings of the Sixteenth International Conference on Machine Learning, p.335-343, June 27-30, 1999
• 18: Robert E. Schapire, A Brief Introduction to Boosting, Proceedings of the Sixteenth International Joint Conference on Artificial Intelligence, p.1401-1406, July 31-August 06, 1999
• 19: Jonathan R Shewchuk, An Introduction to the Conjugate Gradient Method Without the Agonizing Pain, Carnegie Mellon University, Pittsburgh, PA, 1994
• 20: SpyOnIt, http://www.spyonit.com/
• 21: WebSpector, http://www.illumix.com/.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.5 On-line Information Services
          Subjects: Web-based services

Additional Classification:
  G. Mathematics of Computing
  G.3 PROBABILITY AND STATISTICS
      Subjects: Statistical computing

General Terms:
Algorithms

Keywords:
URL monitoring, classification, information extraction, intelligent crawling, machine learning