Discovering unexpected information from your competitors' web sites

ABSTRACT

Ever since the beginning of the Web, finding useful information from the Web has been an important problem. Existing approaches include keyword-based search, wrapper-based information extraction, Web query and user preferences. These approaches essentially find information that matches the user's explicit specifications. This paper argues that this is insufficient. There is another type of information that is also of great interest, i.e., unexpected information, which is unanticipated by the user. Finding unexpected information is useful in many applications. For example, it is useful for a company to find unexpected information bout its competitors, e.g., unexpected services and products that its competitors offer. With this information, the company can learn from its competitors and/or design counter measures to improve its competitiveness. Since the number of pages of a typical commercial site is very large and there are also many relevant sites (competitors), it is very difficult for a human user to view each page to discover the unexpected information. Automated assistance is needed. In this paper, we propose a number of methods to help the user find various types of unexpected information from his/her competitors' Web sites. Experiment results show that these techniques are very useful in practice and also efficient.

REFERENCES

• 1: Rakesh Agrawal , Ramakrishnan Srikant, Fast Algorithms for Mining Association Rules in Large Databases, Proceedings of the 20th International Conference on Very Large Data Bases, p.487-499, September 12-15, 1994
• 2: Naveen Ashish , Craig A. Knoblock, Wrapper generation for semi-structured Internet sources, ACM SIGMOD Record, v.26 n.4, p.8-15, Dec. 1997 [doi>10.1145/271074.271078]
• 3: Ricardo A. Baeza-Yates , Berthier Ribeiro-Neto, Modern Information Retrieval, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1999
• 4: Roberto J. Bayardo, Jr., Efficiently mining long patterns from databases, Proceedings of the 1998 ACM SIGMOD international conference on Management of data, p.85-93, June 01-04, 1998, Seattle, Washington, United States
• 5: Sergey Brin , Lawrence Page, The anatomy of a large-scale hypertextual Web search engine, Proceedings of the seventh international conference on World Wide Web 7, p.107-117, April 1998, Brisbane, Australia
• 6: S. Ceri , S. Comai , E. Damiani , P. Fraternali , L. Tanca, Complex queries in XML-GL, Proceedings of the 2000 ACM symposium on Applied computing, p.888-893, March 2000, Como, Italy [doi>10.1145/338407.338677]
• 7: Soumen Chakrabarti , Martin van den Berg , Byron Dom, Focused crawling: a new approach to topic-specific Web resource discovery, Proceeding of the eighth international conference on World Wide Web, p.1623-1640, May 1999, Toronto, Canada
• 8: William W. Cohen, Integration of heterogeneous databases without common domains using queries based on textual similarity, Proceedings of the 1998 ACM SIGMOD international conference on Management of data, p.201-212, June 01-04, 1998, Seattle, Washington, United States
• 9: Mark Craven , Dan DiPasquo , Dayne Freitag , Andrew McCallum , Tom Mitchell , Kamal Nigam , Seán Slattery, Learning to construct knowledge bases from the World Wide Web, Artificial Intelligence, v.118 n.1-2, p.69-113, April 2000 [doi>10.1016/S0004-3702(00)00004-7]
• 10: Jeffrey Dean , Monika R. Henzinger, Finding related pages in the World Wide Web, Proceeding of the eighth international conference on World Wide Web, p.1467-1479, May 1999, Toronto, Canada
• 11: Dulin Core Home Page, http://purl.org/DC.
• 12: Daniela Florescu , Alon Levy , Alberto Mendelzon, Database techniques for the World-Wide Web: a survey, ACM SIGMOD Record, v.27 n.3, p.59-74, Sept. 1, 1998 [doi>10.1145/290593.290605]
• 13: Ronen Feldman , Yair Liberzon , Binyamin Rosenfeld , Jonathan Schler , Jonathan Stoppi, A framework for specifying explicit bias for revision of approximate information extraction rules, Proceedings of the sixth ACM SIGKDD international conference on Knowledge discovery and data mining, p.189-197, August 20-23, 2000, Boston, Massachusetts, United States [doi>10.1145/347090.347125]
• 14: David Gibson , Jon Kleinberg , Prabhakar Raghavan, Inferring Web communities from link topology, Proceedings of the ninth ACM conference on Hypertext and hypermedia : links, objects, time and space---structure in hypermedia systems: links, objects, time and space---structure in hypermedia systems, p.225-234, June 20-24, 1998, Pittsburgh, Pennsylvania, United States [doi>10.1145/276627.276652]
• 15: Tao Guan , Kam-Fai Wong, KPS: a Web information mining algorithm, Proceeding of the eighth international conference on World Wide Web, p.1495-1507, May 1999, Toronto, Canada
• 16: Jon M. Kleinberg, Authoritative sources in a hyperlinked environment, Proceedings of the ninth annual ACM-SIAM symposium on Discrete algorithms, p.668-677, January 25-27, 1998, San Francisco, California, United States
• 17: B. Liu, and W. Hsu. Post-analysis of learnt rules. AAAI-96.
• 18: B. Liu, W. Hsu, and S. Chen. Using general impressions to analyze discovered classification rules. KDD-97, 1997.
• 19: A. Mendelzon, G. Mihaila, T. Milo. Querying the World Wide Web, Journal of Digital Libraries 1(1): 68-88, 1997.
• 20: Balaji Padmanabhan , Alexander Tuzhilin, Small is beautiful: discovering the minimal set of unexpected patterns, Proceedings of the sixth ACM SIGKDD international conference on Knowledge discovery and data mining, p.54-63, August 20-23, 2000, Boston, Massachusetts, United States [doi>10.1145/347090.347103]
• 21: G. Piatesky-Shapiro & C. Matheus. The interestingness of deviations. KDD-94, 1994.
• 22: Resource Description Framework (RDF) Schema Specification, W3C proposed recommendation. 22 Feb, 1999. http://www, w3.org/TR/PR-rdf-schema/
• 23: Gerard Salton , Michael J. McGill, Introduction to Modern Information Retrieval, McGraw-Hill, Inc., New York, NY, 1986
• 24: Gerard Salton , Christopher Buckley, Term-weighting approaches in automatic text retrieval, Information Processing and Management: an International Journal, v.24 n.5, p.513-523, 1988 [doi>10.1016/0306-4573(88)90021-0]
• 25: Avi Silberschatz , Alexander Tuzhilin, What Makes Patterns Interesting in Knowledge Discovery Systems, IEEE Transactions on Knowledge and Data Engineering, v.8 n.6, p.970-974, December 1996 [doi>10.1109/69.553165]
• 26: George M. Underwood , Paul P. Maglio , Rob Barrett, User-centered push for timely information delivery, Proceedings of the seventh international conference on World Wide Web 7, p.33-41, April 1998, Brisbane, Australia

INDEX TERMS

Primary Classification:
  F. Theory of Computation
  F.2 ANALYSIS OF ALGORITHMS AND PROBLEM COMPLEXITY
      F.2.2 Nonnumerical Algorithms and Problems
          Subjects: Sorting and searching

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.5 On-line Information Services
          Subjects: Web-based services

  J. Computer Applications
  J.1 ADMINISTRATIVE DATA PROCESSING
      Subjects: Business

General Terms:
Algorithms, Design, Performance

Keywords:
Information interestingness, Web comparison, Web mining