Mining product reputations on the Web

ABSTRACT

Knowing the reputations of your own and/or competitors' products is important for marketing and customer relationship management. It is, however, very costly to collect and analyze survey data manually. This paper presents a new framework for mining product reputations on the Internet. It automatically collects people's opinions about target products from Web pages, and it uses text mining techniques to obtain the reputations of those products.On the basis of human-test samples, we generate in advance syntactic and linguistic rules to determine whether any given statement is an opinion or not, as well as whether such any opinion is positive or negative in nature. We first collect statements regarding target products using a general search engine, and then, using the rules, extract opinions from among them and attach three labels to each opinion, labels indicating the positive/negative determination, the product name itself, and an numerical value expressing the degree of system confidence that the statement is, in fact, an opinion. The labeled opinions are then input into an opinion database.The mining of reputations, i.e., the finding of statistically meaningful information included in the database, is then conducted. We specify target categories using label values (such as positive opinions of product A) and perform four types of text mining: extraction of 1) characteristic words, 2) co-occurrence words, 3) typical sentences, for individual target categories, and 4) correspondence analysis among multiple target categories.Actual marketing data is used to demonstrate the validity and effectiveness of the framework, which offers a drastic reduction in the overall cost of reputation analysis over that of conventional survey approaches and supports the discovery of knowledge from the pool of opinions on the web.

REFERENCES

• 1: Brad Adelberg, NoDoSE—a tool for semi-automatically extracting structured and semistructured data from text documents, Proceedings of the 1998 ACM SIGMOD international conference on Management of data, p.283-294, June 01-04, 1998, Seattle, Washington, United States
• 2: Rakesh Agrawal , Ramakrishnan Srikant, Fast Algorithms for Mining Association Rules in Large Databases, Proceedings of the 20th International Conference on Very Large Data Bases, p.487-499, September 12-15, 1994
• 3: M.R. Anderberg, Cluster Analysis for Applications, Academic Press, 1973.
• 4: Naveen Ashish , Craig A. Knoblock, Wrapper generation for semi-structured Internet sources, ACM SIGMOD Record, v.26 n.4, p.8-15, Dec. 1997 [doi>10.1145/271074.271078]
• 5: J.P. Benzecri, Correspondence Analysis Handbook, Mercel Dekker, 1992.
• 6: V. Chaudhri and R. Fikes, Answering Systems, the 1999 Fall Symposium. Technical Report, FS-98-04, AAAI, November 1999.
• 7: David Clark, Technology News, Computer, v.33 n.2, p.18-21, February 2000 [doi>10.1109/MC.2000.820034]
• 8: Mark Craven , Dan DiPasquo , Dayne Freitag , Andrew McCallum , Tom Mitchell , Kamal Nigam , Seán Slattery, Learning to construct knowledge bases from the World Wide Web, Artificial Intelligence, v.118 n.1-2, p.69-113, April 2000 [doi>10.1016/S0004-3702(00)00004-7]
• 9: Robert B. Doorenbos , Oren Etzioni , Daniel S. Weld, A scalable comparison-shopping agent for the World-Wide Web, Proceedings of the first international conference on Autonomous agents, p.39-48, February 05-08, 1997, Marina del Rey, California, United States [doi>10.1145/267658.267666]
• 10: Daniela Florescu , Alon Levy , Alberto Mendelzon, Database techniques for the World-Wide Web: a survey, ACM SIGMOD Record, v.27 n.3, p.59-74, Sept. 1, 1998 [doi>10.1145/290593.290605]
• 11: Fujitsu, Symfoware World http://www.fujitsu.co.jp/jp/soft/symfoware/index.html, 2001.
• 12: Sanda M. Harabagiu , Marius A. Paşca , Steven J. Maiorano, Experiments with open-domain textual Question Answering, Proceedings of the 18th conference on Computational linguistics, p.292-298, July 31-August 04, 2000, Saarbrücken, Germany [doi>10.3115/990820.990863]
• 13: B. Katz, From sentence processing to information access on the World Wide Web. in Natural Language Processing for the World Wide Web: the 1997 AAAI Spring Symposium, pp:77--94, 1999.
• 14: Komatsu Soft, Information Mining Tool VextSearch (in Japanese) http://www.komatsusoft.co.jp/develp/vxtsc/index.html, 2001.
• 15: Hang Li , Kenji Yamanishi, Mining from open answers in questionnaire data, Proceedings of the seventh ACM SIGKDD international conference on Knowledge discovery and data mining, p.443-449, August 26-29, 2001, San Francisco, California [doi>10.1145/502512.502579]
• 16: Hang Li , Kenji Yamanishi, Text classification using ESC-based stochastic decision lists, Information Processing and Management: an International Journal, v.38 n.3, p.343-361, May 2002 [doi>10.1016/S0306-4573(01)00038-3]
• 17: K.C. Litkowski, Question-answering using semantic relation triples.in Proc. of the 8th Text Retrieval Conference (TREC-8)., pp:349--356, 1999.
• 18: Dan Moldovan , Sanda Harabagiu , Marius Pasca , Rada Mihalcea , Roxana Girju , Richard Goodrum , Vasile Rus, The structure and performance of an open-domain question answering system, Proceedings of the 38th Annual Meeting on Association for Computational Linguistics, p.563-570, October 03-06, 2000, Hong Kong [doi>10.3115/1075218.1075289]
• 19: John Prager , Eric Brown , Anni Coden , Dragomir Radev, Question-answering by predictive annotation, Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, p.184-191, July 24-28, 2000, Athens, Greece [doi>10.1145/345508.345574]
• 20: J. Ross Quinlan, C4.5: programs for machine learning, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1993
• 21: J. Rissanen, Fisher information and stochastic complexity, IEEE Transaction on Information Theory, 42(1), pp:40--47, 1996.
• 22: D. R. Radev, J. Prager, and V. Samn, The use of predictive annotation for question answering in Proc. of the 8th Text Retrieval Conference (TREC-8), pp:399--411, 1999.
• 23: R. Srihari and W. Li, Information extraction supported question answering, in Proc. of the 8th Text Retrieval Conference (TREC-8), pp:185--196, 1999.
• 24: K. Tateishi, Y. Ishiguro, and T. Fukushima, A reputation search engine that gathers people's opinions from the internet, (in Japanese) Technical Report NL-144-11, Information Processing Society of Japan, pp:75--82, 2001.
• 25: Ellen M. Voorhees , Dawn M. Tice, Building a question answering test collection, Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, p.200-207, July 24-28, 2000, Athens, Greece [doi>10.1145/345508.345577]
• 26: Kenji Yamanishi, A Learning Criterion for Stochastic Rules, Machine Learning, v.9 n.2-3, p.165-203, July 1992 [doi>10.1007/BF00992676]
• 27: K. Yamanishi, A decision-theoretic extension of stochastic complexity and its applications to learning, IEEE Trans. on Infortmation Theory, 44(4), pp:1424--1439, 1998.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.2 DATABASE MANAGEMENT
      H.2.8 Database applications
          Subjects: Data mining

Additional Classification:
  G. Mathematics of Computing
  G.1 NUMERICAL ANALYSIS
      G.1.6 Optimization
          Subjects: Stochastic programming
  G.3 PROBABILITY AND STATISTICS
      Subjects: Statistical computing

General Terms:
Algorithms, Design, Experimentation, Performance